2024-03-29T10:57:42.421 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:57:42.421 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/orig.config.yaml 2024-03-29T10:57:42.457 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:57:42.524 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622675' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGe2cbzlf1n6FisHkjzCBQV8qL7Y7EdolAUPvoVYE70dLHfL29aFbDMXR0dueLZBqTXP40Xs8fCa9gEC2xFZUw= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2KushoYANTljforGMe/c4ZdrBe6UZpibQLuMPDjQiS3b4uSst/CWLgNi/sc8NskVETh48c5oBQyYnNV/cicqo= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:57:42.524 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:57:42.525 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:57:42.525 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:57:42.526 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:57:42.550 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:57:42.550 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:57:42.551 INFO:teuthology.packaging:ref: None 2024-03-29T10:57:42.551 INFO:teuthology.packaging:tag: None 2024-03-29T10:57:42.551 INFO:teuthology.packaging:branch: main 2024-03-29T10:57:42.551 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:57:42.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:57:42.854 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:57:42.855 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:57:42.865 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:57:42.865 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:57:42.891 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:57:42.904 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:57:42.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:57:42.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:48.700274', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGe2cbzlf1n6FisHkjzCBQV8qL7Y7EdolAUPvoVYE70dLHfL29aFbDMXR0dueLZBqTXP40Xs8fCa9gEC2xFZUw='} 2024-03-29T10:57:42.976 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:48.701671', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2KushoYANTljforGMe/c4ZdrBe6UZpibQLuMPDjQiS3b4uSst/CWLgNi/sc8NskVETh48c5oBQyYnNV/cicqo='} 2024-03-29T10:57:42.977 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:57:42.985 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:57:42.986 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:57:42.986 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:57:43.069 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1cfe5bb0d0>, signals=[15]) 2024-03-29T10:57:43.069 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:57:43.081 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:57:43.081 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T10:57:43.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:57:43.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T10:57:43.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:57:43.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:57:43.283 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-03-29T10:57:43.301 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-03-29T10:57:43.301 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-03-29T10:57:43.357 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-03-29T10:57:43.357 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-03-29T10:57:43.357 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:57:43.358 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:57:43.359 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-03-29T10:57:43.385 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-03-29T10:57:43.401 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-03-29T10:57:43.401 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-03-29T10:57:43.457 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-03-29T10:57:43.457 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-03-29T10:57:43.457 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-03-29T10:57:43.457 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:57:43.457 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:57:43.458 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:57:43.458 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-03-29T10:57:43.476 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:57:43.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:57:43.541 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:57:43.541 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:57:43.543 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:57:43.558 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:57:43.569 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:57:43.569 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:57:43.600 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:57:43.695 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:57:43.754 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:57:43.773 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:57:43.774 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:57:43.774 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:57:43.774 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:57:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:43.789 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-29T10:57:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:43.791 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-03-29T10:57:43.846 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:57:43.846 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-03-29T10:57:43.846 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-03-29T10:57:43.847 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:57:43.847 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-361.el9.x86_64 2024-03-29T10:57:43.848 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-03-29T10:57:45.849 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:57:45.983 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:57:47.516 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T10:57:47.576 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-29T10:57:48.298 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-29T10:57:50.808 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T10:57:56.982 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 891 kB/s | 8.0 MB 00:09 2024-03-29T10:57:59.333 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-29T10:58:02.091 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 808 kB/s | 6.1 MB 00:07 2024-03-29T10:58:03.666 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-29T10:58:04.509 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-29T10:58:05.310 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T10:58:05.926 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T10:58:07.036 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-29T10:58:10.184 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T10:58:12.053 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T10:58:12.458 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:58:12.459 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:58:12.561 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:58:12.568 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:58:12.569 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 76 M 2024-03-29T10:58:12.570 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 123 M 2024-03-29T10:58:12.570 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T10:58:12.711 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:58:13.311 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T10:58:13.778 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:58:14.170 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T10:58:14.173 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:14.174 INFO:teuthology.orchestra.run.smithi110.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-29T10:58:14.174 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T10:58:14.326 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:58:14.326 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:58:14.440 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T10:58:14.450 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T10:58:14.450 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-03-29T10:58:14.451 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:58:14.452 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 76 M 2024-03-29T10:58:14.452 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 123 M 2024-03-29T10:58:14.452 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T10:58:14.551 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T10:58:14.551 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T10:58:14.596 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:58:15.279 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T10:58:15.430 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T10:58:15.430 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T10:58:16.029 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-29T10:58:16.321 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T10:58:16.324 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:16.325 INFO:teuthology.orchestra.run.smithi105.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-29T10:58:16.325 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T10:58:16.686 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T10:58:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T10:58:17.512 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T10:58:17.513 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T10:58:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T10:58:18.395 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:58:18.489 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:58:19.807 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T10:58:20.403 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:58:20.488 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:58:21.209 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:58:21.342 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:58:23.238 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:58:23.383 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:58:26.266 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:58:26.566 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:58:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:58:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:58:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:58:32.822 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:00.433 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:00.433 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:59:00.433 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:59:00.476 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:00.823 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:00.823 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:00.823 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:00.823 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:01.235 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:01.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:59:01.235 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T10:59:01.235 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.235 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.236 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.236 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:59:01.236 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T10:59:01.577 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:01.669 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.669 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:01.670 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:59:01.670 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:59:01.670 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:59:01.670 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:59:01.670 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:59:01.670 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:01.751 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:01.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:01.752 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-03-29T10:59:01.793 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:59:01.794 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:59:01.795 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:59:01.795 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-03-29T10:59:01.795 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T10:59:01.795 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:59:01.796 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:59:01.796 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:59:01.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:59:02.139 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:02.139 INFO:teuthology.orchestra.run.smithi105.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:59:02.139 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:59:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:02.392 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-03-29T10:59:02.513 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:02.513 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:02.513 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:02.514 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:59:02.918 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T10:59:03.243 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:03.349 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:03.349 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:03.349 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:59:03.349 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:59:03.349 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:59:03.350 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:59:03.350 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:59:03.350 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:03.430 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:03.430 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:03.430 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-03-29T10:59:03.464 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-03-29T10:59:03.464 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-03-29T10:59:03.464 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-03-29T10:59:03.464 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 68.el9 2024-03-29T10:59:03.464 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8264254 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:59:03.465 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:59:03.466 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:59:03.466 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:59:03.466 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-03-29T10:59:03.466 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T10:59:03.466 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:59:03.467 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:59:03.467 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:59:03.467 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:59:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:59:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-03-29T10:59:03.887 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-29T10:59:03.901 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.SRt57JoK0I 2024-03-29T10:59:03.902 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SRt57JoK0I 2024-03-29T10:59:03.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.SRt57JoK0I 2024-03-29T10:59:04.058 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-03-29T10:59:04.107 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.SRt57JoK0I is 6KB 2024-03-29T10:59:04.156 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.SRt57JoK0I 2024-03-29T10:59:04.169 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:59:04.241 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:59:04.241 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:59:04.241 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:59:04.241 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:59:04.243 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:59:04.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-03-29T10:59:05.436 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:59:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-03-29T10:59:05.468 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-03-29T10:59:05.483 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.YCqaxUvB3a 2024-03-29T10:59:05.483 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YCqaxUvB3a 2024-03-29T10:59:05.552 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.YCqaxUvB3a 2024-03-29T10:59:05.694 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.YCqaxUvB3a is 6KB 2024-03-29T10:59:05.742 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.YCqaxUvB3a 2024-03-29T10:59:05.756 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:59:05.829 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:59:05.829 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:59:05.829 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:59:05.830 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:59:05.831 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:59:05.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-03-29T10:59:34.446 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:59:34.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T10:59:34.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:59:35.976 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:59:35.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T10:59:35.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:59:52.991 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-29T10:59:54.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-29T11:00:01.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T11:00:01.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:03.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T11:00:03.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:06.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-29T11:00:18.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T11:00:18.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:50.460 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-03-29T11:00:50.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T11:00:50.817 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:00:50.817 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-03-29T11:00:50.831 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:00:50.831 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:00:50.831 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:00:50.831 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:00:51.833 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:00:51.833 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:00:51.907 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-03-29T11:00:51.969 DEBUG:teuthology.parallel:result is None 2024-03-29T11:01:01.996 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:01:13.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T11:01:13.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:14.273 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-29T11:01:14.672 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T11:01:14.672 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:01:14.672 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-29T11:01:14.687 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:01:14.687 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:01:14.687 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:01:14.687 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:01:15.688 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:01:15.689 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:01:15.780 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-03-29T11:01:15.818 DEBUG:teuthology.parallel:result is None 2024-03-29T11:01:15.818 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:01:15.826 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:01:15.826 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:01:15.829 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:01:15.847 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:01:15.854 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:01:15.869 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:01:15.869 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:01:15.887 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:01:15.944 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:01:15.950 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:01:15.950 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:01:15.953 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:01:15.984 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:15.992 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:15.998 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:16.007 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:16.009 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:01:16.017 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:01:16.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:01:16.042 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:01:16.079 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:01:16.088 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:01:16.088 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:01:16.112 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:01:16.135 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:16.202 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:16.258 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:16.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:16.385 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:01:16.385 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:01:16.452 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:16.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:16.562 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:16.642 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:16.691 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:01:16.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:01:16.761 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-03-29T11:01:16.763 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-03-29T11:01:16.807 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:01:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:01:17.233 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:01:17.242 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:01:17.242 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:01:17.253 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:01:17.286 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-03-29T11:01:17.353 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-03-29T11:01:17.355 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-03-29T11:01:17.416 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-03-29T11:01:17.418 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:01:17.418 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-03-29T11:01:17.444 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-03-29T11:01:17.445 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-03-29T11:01:17.462 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-03-29T11:01:17.462 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:01:17.462 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:01:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:17.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:01:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:17.546 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:01:17.547 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:01:17.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:01:17.630 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:01:17.640 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:01:17.645 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:01:17.794 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:01:17.809 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:01:17.810 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:05:32.451 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2024-03-29T11:05:32.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T11:05:32.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:32.546 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-03-29T11:05:32.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-29T11:05:32.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T11:05:32.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:32.670 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-29T11:05:32.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-29T11:05:32.745 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:05:32.755 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:05:32.755 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:05:32.755 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:05:32.758 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:05:32.758 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:05:32.786 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:05:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:05:32.816 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:05:32.831 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-03-29T11:05:32.833 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:05:32.845 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-03-29T11:05:32.861 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:05:32.862 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-03-29T11:05:32.875 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-03-29T11:05:32.877 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:05:32.892 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:05:32.908 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:05:32.927 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:05:32.930 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:05:32.930 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-03-29T11:05:32.931 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.931 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.931 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.931 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-03-29T11:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:05:32.963 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:05:32.971 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:05:32.971 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:05:32.971 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:05:32.971 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:05:32.977 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:05:32.977 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:05:32.977 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:05:32.978 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:05:32.980 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:05:32.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:05:33.125 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:05:33.125 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:05:33.130 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:05:33.130 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:05:33.252 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:05:33.253 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:05:33.253 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:05:33.281 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:05:33.281 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:05:33.334 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:05:33.334 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:05:33.334 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:05:33.349 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:05:33.362 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:05:33.363 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:05:33.430 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:05:33.431 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:05:33.496 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-03-29T11:05:33.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-03-29T11:05:33.513 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:05:33.579 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-03-29T11:05:33.581 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-03-29T11:05:33.746 INFO:teuthology.orchestra.run.smithi110.stdout:74 files removed 2024-03-29T11:05:33.773 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-03-29T11:05:33.818 INFO:teuthology.orchestra.run.smithi105.stdout:74 files removed 2024-03-29T11:05:33.847 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-03-29T11:05:34.476 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 203 kB/s | 82 kB 00:00 2024-03-29T11:05:34.531 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 212 kB/s | 82 kB 00:00 2024-03-29T11:05:34.835 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-29T11:05:34.915 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-29T11:05:35.240 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-29T11:05:35.244 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T11:05:35.685 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:05:35.688 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:05:37.135 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-29T11:05:37.272 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-29T11:05:37.968 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T11:05:38.087 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-29T11:05:39.693 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-29T11:05:45.596 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-29T11:05:49.522 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-29T11:05:50.890 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-29T11:05:51.578 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 745 kB/s | 6.1 MB 00:08 2024-03-29T11:05:52.600 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:05:52.970 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-29T11:05:55.130 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-29T11:05:57.591 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T11:06:00.055 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:06:00.188 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:00.198 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:00.198 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:06:00.198 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:00.198 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:00.198 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:06:00.199 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse 2024-03-29T11:06:00.200 INFO:teuthology.orchestra.run.smithi110.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-03-29T11:06:00.201 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout:Install 30 Packages 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-03-29T11:06:00.202 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:00.203 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 58 M 2024-03-29T11:06:00.203 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:01.609 INFO:teuthology.orchestra.run.smithi110.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.3 MB/s | 9.7 MB 00:01 2024-03-29T11:06:01.651 INFO:teuthology.orchestra.run.smithi110.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 609 kB/s | 25 kB 00:00 2024-03-29T11:06:01.784 INFO:teuthology.orchestra.run.smithi110.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T11:06:01.876 INFO:teuthology.orchestra.run.smithi110.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T11:06:02.185 INFO:teuthology.orchestra.run.smithi110.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-29T11:06:02.235 INFO:teuthology.orchestra.run.smithi110.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 907 kB/s | 45 kB 00:00 2024-03-29T11:06:02.302 INFO:teuthology.orchestra.run.smithi110.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:06:02.369 INFO:teuthology.orchestra.run.smithi110.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T11:06:02.444 INFO:teuthology.orchestra.run.smithi110.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.0 MB/s | 5.1 MB 00:00 2024-03-29T11:06:02.478 INFO:teuthology.orchestra.run.smithi110.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.9 MB/s | 320 kB 00:00 2024-03-29T11:06:02.503 INFO:teuthology.orchestra.run.smithi110.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:06:02.520 INFO:teuthology.orchestra.run.smithi110.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T11:06:02.648 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:06:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:06:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-03-29T11:06:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:Install 30 Packages 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 58 M 2024-03-29T11:06:02.655 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:06:02.704 INFO:teuthology.orchestra.run.smithi110.stdout:(13/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 5.3 MB 00:02 2024-03-29T11:06:02.729 INFO:teuthology.orchestra.run.smithi110.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-03-29T11:06:02.754 INFO:teuthology.orchestra.run.smithi110.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 159 kB 00:00 2024-03-29T11:06:02.771 INFO:teuthology.orchestra.run.smithi110.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-29T11:06:02.896 INFO:teuthology.orchestra.run.smithi110.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-29T11:06:02.922 INFO:teuthology.orchestra.run.smithi110.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-29T11:06:02.938 INFO:teuthology.orchestra.run.smithi110.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 246 kB/s | 41 kB 00:00 2024-03-29T11:06:02.964 INFO:teuthology.orchestra.run.smithi110.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:06:02.981 INFO:teuthology.orchestra.run.smithi110.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 843 kB/s | 35 kB 00:00 2024-03-29T11:06:02.997 INFO:teuthology.orchestra.run.smithi110.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 798 kB/s | 46 kB 00:00 2024-03-29T11:06:03.298 INFO:teuthology.orchestra.run.smithi110.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 87 kB/s | 26 kB 00:00 2024-03-29T11:06:03.440 INFO:teuthology.orchestra.run.smithi110.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 650 kB/s | 309 kB 00:00 2024-03-29T11:06:03.473 INFO:teuthology.orchestra.run.smithi110.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-29T11:06:03.507 INFO:teuthology.orchestra.run.smithi110.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:06:03.741 INFO:teuthology.orchestra.run.smithi110.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 821 kB/s | 192 kB 00:00 2024-03-29T11:06:03.866 INFO:teuthology.orchestra.run.smithi110.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-29T11:06:03.914 INFO:teuthology.orchestra.run.smithi105.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.1 MB/s | 5.3 MB 00:01 2024-03-29T11:06:03.972 INFO:teuthology.orchestra.run.smithi105.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 434 kB/s | 25 kB 00:00 2024-03-29T11:06:04.106 INFO:teuthology.orchestra.run.smithi105.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.8 MB/s | 9.7 MB 00:01 2024-03-29T11:06:04.156 INFO:teuthology.orchestra.run.smithi105.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.9 MB/s | 742 kB 00:00 2024-03-29T11:06:04.208 INFO:teuthology.orchestra.run.smithi105.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.8 MB/s | 498 kB 00:00 2024-03-29T11:06:04.224 INFO:teuthology.orchestra.run.smithi110.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-03-29T11:06:04.282 INFO:teuthology.orchestra.run.smithi105.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 611 kB/s | 45 kB 00:00 2024-03-29T11:06:04.366 INFO:teuthology.orchestra.run.smithi105.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:06:04.458 INFO:teuthology.orchestra.run.smithi105.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T11:06:04.575 INFO:teuthology.orchestra.run.smithi105.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.7 MB/s | 320 kB 00:00 2024-03-29T11:06:04.758 INFO:teuthology.orchestra.run.smithi105.stdout:(10/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 10 MB/s | 20 MB 00:01 2024-03-29T11:06:04.783 INFO:teuthology.orchestra.run.smithi105.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 298 kB 00:00 2024-03-29T11:06:04.809 INFO:teuthology.orchestra.run.smithi105.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T11:06:04.858 INFO:teuthology.orchestra.run.smithi110.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.9 MB/s | 3.1 MB 00:00 2024-03-29T11:06:04.926 INFO:teuthology.orchestra.run.smithi105.stdout:(13/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 5.1 MB 00:00 2024-03-29T11:06:04.991 INFO:teuthology.orchestra.run.smithi110.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:01 2024-03-29T11:06:05.012 INFO:teuthology.orchestra.run.smithi105.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-29T11:06:05.018 INFO:teuthology.orchestra.run.smithi105.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 762 kB/s | 159 kB 00:00 2024-03-29T11:06:05.043 INFO:teuthology.orchestra.run.smithi105.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 382 kB/s | 44 kB 00:00 2024-03-29T11:06:05.050 INFO:teuthology.orchestra.run.smithi110.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.9 MB/s | 3.4 MB 00:01 2024-03-29T11:06:05.057 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:05.057 INFO:teuthology.orchestra.run.smithi110.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T11:06:05.058 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:06:05.095 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:06:05.095 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:06:05.252 INFO:teuthology.orchestra.run.smithi105.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 661 kB 00:00 2024-03-29T11:06:05.277 INFO:teuthology.orchestra.run.smithi105.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 176 kB/s | 41 kB 00:00 2024-03-29T11:06:05.302 INFO:teuthology.orchestra.run.smithi105.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-03-29T11:06:05.327 INFO:teuthology.orchestra.run.smithi105.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 220 kB 00:00 2024-03-29T11:06:05.353 INFO:teuthology.orchestra.run.smithi105.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 466 kB/s | 35 kB 00:00 2024-03-29T11:06:05.378 INFO:teuthology.orchestra.run.smithi105.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 618 kB/s | 46 kB 00:00 2024-03-29T11:06:05.572 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:06:05.572 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:06:05.686 INFO:teuthology.orchestra.run.smithi105.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 84 kB/s | 26 kB 00:00 2024-03-29T11:06:05.812 INFO:teuthology.orchestra.run.smithi105.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 673 kB/s | 309 kB 00:00 2024-03-29T11:06:05.837 INFO:teuthology.orchestra.run.smithi105.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2024-03-29T11:06:05.895 INFO:teuthology.orchestra.run.smithi105.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2024-03-29T11:06:06.004 INFO:teuthology.orchestra.run.smithi105.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-29T11:06:06.113 INFO:teuthology.orchestra.run.smithi105.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-03-29T11:06:06.155 INFO:teuthology.orchestra.run.smithi105.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-29T11:06:06.297 INFO:teuthology.orchestra.run.smithi105.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-03-29T11:06:06.547 INFO:teuthology.orchestra.run.smithi105.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.8 MB/s | 3.4 MB 00:00 2024-03-29T11:06:06.597 INFO:teuthology.orchestra.run.smithi105.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-03-29T11:06:06.606 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:06.606 INFO:teuthology.orchestra.run.smithi105.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:06:06.606 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:06:06.638 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:06:06.638 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:06:07.014 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:06:07.100 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:06:07.112 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:06:07.112 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:06:07.129 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:06:07.155 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:06:07.186 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:07.203 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:07.361 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:07.416 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:07.751 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:06:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:07.971 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:06:08.000 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:08.062 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:08.091 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:06:08.135 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:06:08.202 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:08.227 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:08.333 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:06:08.441 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:06:08.468 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:06:08.513 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:06:08.550 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:06:08.578 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:06:08.608 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:06:08.611 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:06:08.687 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:06:08.726 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:06:08.752 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:06:08.784 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:08.800 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:08.929 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:06:08.958 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:08.983 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:09.039 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:09.047 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:09.079 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:06:09.123 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:06:09.359 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:06:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:09.511 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:09.569 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:06:09.599 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:06:09.602 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:09.630 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:09.676 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:09.682 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:09.705 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:06:09.748 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:06:09.749 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:06:09.779 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:06:09.816 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:09.840 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:06:10.055 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:06:10.090 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:06:10.131 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:06:10.165 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:06:10.192 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:06:10.225 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:06:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:06:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:10.652 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:10.684 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:06:10.728 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:06:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:06:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:11.262 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:11.329 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:06:11.360 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:06:11.834 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:11.876 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:12.426 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:12.462 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:12.539 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:12.539 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:06:12.539 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:06:12.539 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:12.580 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:13.573 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:13.623 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:14.238 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:14.396 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:14.396 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:06:14.396 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:06:14.396 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:14.436 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-03-29T11:06:22.278 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-03-29T11:06:22.279 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:23.217 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:23.241 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:23.241 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:23.242 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:06:23.242 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:23.242 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:23.242 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:23.243 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:23.327 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:23.328 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:24.043 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:24.043 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-03-29T11:06:24.043 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-03-29T11:06:24.043 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-03-29T11:06:24.044 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-03-29T11:06:24.044 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-03-29T11:06:24.044 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-03-29T11:06:24.044 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-03-29T11:06:24.044 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:24.460 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:24.460 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:06:24.460 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:06:24.460 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:06:24.460 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:06:24.461 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:06:24.462 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:06:24.463 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:06:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:24.974 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:24.974 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:06:24.974 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:24.974 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:25.060 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:25.060 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:25.402 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:25.402 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:25.402 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-03-29T11:06:25.402 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.402 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.403 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:25.404 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:25.405 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:25.620 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-03-29T11:06:26.172 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:26.172 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:06:26.172 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:06:26.173 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:06:26.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:06:26.175 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:06:26.175 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:26.175 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:06:26.175 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:06:26.176 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:26.331 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:06:26.332 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-03-29T11:06:26.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:26.334 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-03-29T11:06:26.334 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 204 M 2024-03-29T11:06:26.334 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:26.559 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:06:26.601 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T11:06:26.626 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T11:06:26.760 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:06:26.785 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 393 kB/s | 163 kB 00:00 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:06:27.161 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.162 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:06:27.163 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:06:27.164 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:06:27.164 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:06:27.164 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:06:27.164 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:27.164 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:27.564 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-03-29T11:06:28.126 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:28.287 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:06:28.288 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 48 M 2024-03-29T11:06:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 204 M 2024-03-29T11:06:28.290 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:06:28.465 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-29T11:06:28.532 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.9 MB/s | 300 kB 00:00 2024-03-29T11:06:28.566 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T11:06:28.666 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 481 kB/s | 163 kB 00:00 2024-03-29T11:06:29.369 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 16 MB/s | 47 MB 00:02 2024-03-29T11:06:29.372 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:29.372 INFO:teuthology.orchestra.run.smithi110.stdout:Total 16 MB/s | 48 MB 00:03 2024-03-29T11:06:29.387 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:06:29.407 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:06:29.407 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:06:29.424 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 218 kB/s | 187 kB 00:00 2024-03-29T11:06:29.791 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:06:29.791 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:06:30.385 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:06:30.452 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:06:30.498 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:06:30.538 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:06:30.586 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:06:30.622 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:06:30.975 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T11:06:30.978 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:30.978 INFO:teuthology.orchestra.run.smithi105.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T11:06:30.992 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:06:31.014 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:06:31.014 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:06:31.398 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:06:31.399 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:06:31.965 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:06:32.033 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:06:32.077 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:06:32.117 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:06:32.158 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:06:32.186 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:06:35.129 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:06:35.214 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:06:36.749 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:06:36.821 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:06:37.592 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:06:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:06:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:06:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:06:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:06:37.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:06:37.953 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:06:37.953 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:37.953 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:37.954 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:38.273 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-03-29T11:06:38.856 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:39.021 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:06:39.024 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:06:39.025 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:06:39.026 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout:Install 36 Packages 2024-03-29T11:06:39.027 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:39.028 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 30 M 2024-03-29T11:06:39.028 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 107 M 2024-03-29T11:06:39.028 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:06:39.100 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:06:39.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:06:39.448 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:39.449 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:39.648 INFO:teuthology.orchestra.run.smithi110.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T11:06:39.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-03-29T11:06:39.757 INFO:teuthology.orchestra.run.smithi110.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-03-29T11:06:40.041 INFO:teuthology.orchestra.run.smithi110.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 2.1 MB 00:00 2024-03-29T11:06:40.141 INFO:teuthology.orchestra.run.smithi110.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T11:06:40.267 INFO:teuthology.orchestra.run.smithi110.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:06:40.296 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:40.300 INFO:teuthology.orchestra.run.smithi110.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-29T11:06:40.325 INFO:teuthology.orchestra.run.smithi110.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:06:40.350 INFO:teuthology.orchestra.run.smithi110.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T11:06:40.376 INFO:teuthology.orchestra.run.smithi110.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T11:06:40.418 INFO:teuthology.orchestra.run.smithi110.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:06:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout:========================================================================================= 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout:========================================================================================= 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:06:40.451 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:06:40.452 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:06:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout:========================================================================================= 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout:Install 36 Packages 2024-03-29T11:06:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:40.455 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 30 M 2024-03-29T11:06:40.455 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 107 M 2024-03-29T11:06:40.455 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:06:40.485 INFO:teuthology.orchestra.run.smithi110.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:06:40.527 INFO:teuthology.orchestra.run.smithi110.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-29T11:06:40.702 INFO:teuthology.orchestra.run.smithi110.stdout:(13/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 18 MB/s | 17 MB 00:00 2024-03-29T11:06:40.727 INFO:teuthology.orchestra.run.smithi110.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 619 kB/s | 124 kB 00:00 2024-03-29T11:06:40.753 INFO:teuthology.orchestra.run.smithi110.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:06:40.828 INFO:teuthology.orchestra.run.smithi110.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-29T11:06:40.962 INFO:teuthology.orchestra.run.smithi110.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 534 kB/s | 125 kB 00:00 2024-03-29T11:06:41.071 INFO:teuthology.orchestra.run.smithi110.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-03-29T11:06:41.130 INFO:teuthology.orchestra.run.smithi105.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T11:06:41.389 INFO:teuthology.orchestra.run.smithi105.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.0 MB/s | 2.1 MB 00:00 2024-03-29T11:06:41.404 INFO:teuthology.orchestra.run.smithi110.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 98 kB/s | 43 kB 00:00 2024-03-29T11:06:41.429 INFO:teuthology.orchestra.run.smithi110.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 251 kB/s | 151 kB 00:00 2024-03-29T11:06:41.521 INFO:teuthology.orchestra.run.smithi110.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-29T11:06:41.622 INFO:teuthology.orchestra.run.smithi110.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-29T11:06:41.647 INFO:teuthology.orchestra.run.smithi110.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 311 kB/s | 172 kB 00:00 2024-03-29T11:06:41.656 INFO:teuthology.orchestra.run.smithi105.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.9 MB/s | 4.7 MB 00:00 2024-03-29T11:06:41.715 INFO:teuthology.orchestra.run.smithi105.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-29T11:06:41.722 INFO:teuthology.orchestra.run.smithi110.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T11:06:41.740 INFO:teuthology.orchestra.run.smithi105.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-29T11:06:41.747 INFO:teuthology.orchestra.run.smithi110.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-29T11:06:41.773 INFO:teuthology.orchestra.run.smithi110.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-29T11:06:41.815 INFO:teuthology.orchestra.run.smithi110.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-29T11:06:41.848 INFO:teuthology.orchestra.run.smithi110.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-03-29T11:06:41.865 INFO:teuthology.orchestra.run.smithi105.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:06:41.882 INFO:teuthology.orchestra.run.smithi105.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-29T11:06:41.890 INFO:teuthology.orchestra.run.smithi110.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 672 kB/s | 79 kB 00:00 2024-03-29T11:06:41.932 INFO:teuthology.orchestra.run.smithi105.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 473 kB/s | 32 kB 00:00 2024-03-29T11:06:41.949 INFO:teuthology.orchestra.run.smithi110.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:06:41.957 INFO:teuthology.orchestra.run.smithi105.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 628 kB/s | 47 kB 00:00 2024-03-29T11:06:41.982 INFO:teuthology.orchestra.run.smithi105.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-29T11:06:42.008 INFO:teuthology.orchestra.run.smithi105.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:06:42.016 INFO:teuthology.orchestra.run.smithi110.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T11:06:42.041 INFO:teuthology.orchestra.run.smithi110.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 597 kB/s | 90 kB 00:00 2024-03-29T11:06:42.058 INFO:teuthology.orchestra.run.smithi105.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:06:42.067 INFO:teuthology.orchestra.run.smithi110.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 304 kB/s | 36 kB 00:00 2024-03-29T11:06:42.083 INFO:teuthology.orchestra.run.smithi105.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:06:42.109 INFO:teuthology.orchestra.run.smithi105.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:06:42.134 INFO:teuthology.orchestra.run.smithi105.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:06:42.150 INFO:teuthology.orchestra.run.smithi110.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T11:06:42.176 INFO:teuthology.orchestra.run.smithi110.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-29T11:06:42.301 INFO:teuthology.orchestra.run.smithi110.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-29T11:06:42.306 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:42.306 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.0 MB/s | 30 MB 00:03 2024-03-29T11:06:42.331 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:06:42.368 INFO:teuthology.orchestra.run.smithi105.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-29T11:06:42.373 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:06:42.374 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:06:42.393 INFO:teuthology.orchestra.run.smithi105.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-29T11:06:42.418 INFO:teuthology.orchestra.run.smithi105.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 404 kB/s | 125 kB 00:00 2024-03-29T11:06:42.685 INFO:teuthology.orchestra.run.smithi105.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 149 kB/s | 43 kB 00:00 2024-03-29T11:06:42.752 INFO:teuthology.orchestra.run.smithi105.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-29T11:06:42.802 INFO:teuthology.orchestra.run.smithi105.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 447 kB/s | 172 kB 00:00 2024-03-29T11:06:42.819 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:06:42.820 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:06:42.827 INFO:teuthology.orchestra.run.smithi105.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T11:06:42.878 INFO:teuthology.orchestra.run.smithi105.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-29T11:06:42.903 INFO:teuthology.orchestra.run.smithi105.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T11:06:42.928 INFO:teuthology.orchestra.run.smithi105.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T11:06:42.953 INFO:teuthology.orchestra.run.smithi105.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-29T11:06:42.979 INFO:teuthology.orchestra.run.smithi105.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T11:06:43.004 INFO:teuthology.orchestra.run.smithi105.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-03-29T11:06:43.046 INFO:teuthology.orchestra.run.smithi105.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 852 kB/s | 79 kB 00:00 2024-03-29T11:06:43.071 INFO:teuthology.orchestra.run.smithi105.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-03-29T11:06:43.105 INFO:teuthology.orchestra.run.smithi105.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-29T11:06:43.130 INFO:teuthology.orchestra.run.smithi105.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T11:06:43.156 INFO:teuthology.orchestra.run.smithi105.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-03-29T11:06:43.189 INFO:teuthology.orchestra.run.smithi105.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T11:06:43.231 INFO:teuthology.orchestra.run.smithi105.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-29T11:06:43.306 INFO:teuthology.orchestra.run.smithi105.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-29T11:06:43.312 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:43.312 INFO:teuthology.orchestra.run.smithi105.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T11:06:43.334 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:06:43.373 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:06:43.373 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:06:43.774 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:06:43.775 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:06:44.135 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:06:44.195 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:06:44.228 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:06:44.289 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:06:44.360 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:06:44.396 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:06:44.429 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:06:44.522 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:06:44.600 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:06:44.640 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:06:44.712 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:06:44.766 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:06:44.812 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:06:44.862 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:06:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:06:45.103 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:06:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:06:45.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:06:45.155 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:06:45.189 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:06:45.195 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:06:45.224 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:06:45.228 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:06:45.253 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:06:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:06:45.331 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:06:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:06:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:06:45.372 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:06:45.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:06:45.459 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:06:45.463 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:06:45.498 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:06:45.529 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:06:45.532 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:06:45.570 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:06:45.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:06:45.642 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:06:45.655 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:06:45.704 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:06:45.728 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:06:45.742 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:06:45.791 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:06:45.846 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:06:45.866 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:06:45.998 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:06:46.037 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:06:46.080 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:06:46.122 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:06:46.153 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:06:46.154 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:06:46.183 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:06:46.185 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:06:46.211 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:46.259 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:06:46.294 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:06:46.328 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:06:46.367 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:06:46.403 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:06:46.437 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:06:46.525 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:06:46.585 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:06:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:06:46.792 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:06:46.919 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:06:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:06:47.123 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:06:47.148 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:47.644 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:06:47.671 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:48.112 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:06:48.135 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:06:48.135 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:48.135 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:06:48.135 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:06:48.136 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:06:48.136 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:48.345 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:06:48.369 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:48.435 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:06:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:06:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:06:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:06:49.090 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:49.090 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:06:49.090 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:06:49.090 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:06:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:06:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:06:49.315 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:49.315 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:06:49.315 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:06:49.315 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:06:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:06:50.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:06:50.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:06:50.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:06:50.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:06:51.123 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:06:51.124 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:06:51.125 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:06:51.231 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:06:51.231 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:51.231 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:06:51.231 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:51.231 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:06:51.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:06:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:51.234 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:51.579 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:06:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:06:52.046 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:06:52.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:06:52.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:06:52.047 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:52.047 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:52.154 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:52.253 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:06:52.306 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:52.308 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:06:52.308 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:52.382 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-03-29T11:06:52.405 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-03-29T11:06:52.941 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:52.995 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:53.094 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:53.094 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:06:53.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:53.096 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:06:53.096 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 747 k 2024-03-29T11:06:53.096 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 754 k 2024-03-29T11:06:53.096 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:53.148 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:53.148 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:06:53.148 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:53.212 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-03-29T11:06:53.525 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:06:53.526 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:53.526 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:06:53.526 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:06:53.531 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:06:53.531 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:06:53.537 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:06:53.537 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:06:53.613 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:06:53.780 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:53.932 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:53.933 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 747 k 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 754 k 2024-03-29T11:06:53.934 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:06:54.050 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:54.130 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:54.217 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T11:06:54.218 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:54.218 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T11:06:54.218 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:06:54.223 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:06:54.223 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:06:54.229 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:06:54.229 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:06:54.303 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:06:54.500 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:54.767 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:54.768 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:54.768 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:06:54.768 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:06:54.768 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:54.768 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:54.825 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:54.914 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:55.056 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:06:55.285 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:55.571 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:55.627 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:55.778 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:06:55.779 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 145 k 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 436 k 2024-03-29T11:06:55.780 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:55.781 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:06:56.007 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 640 kB/s | 145 kB 00:00 2024-03-29T11:06:56.008 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:56.008 INFO:teuthology.orchestra.run.smithi110.stdout:Total 636 kB/s | 145 kB 00:00 2024-03-29T11:06:56.008 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:06:56.016 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:06:56.016 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:06:56.068 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:06:56.068 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:06:56.210 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:06:56.267 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:56.291 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:56.291 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:56.291 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:06:56.291 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:56.346 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:56.501 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:56.501 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:06:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 145 k 2024-03-29T11:06:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 436 k 2024-03-29T11:06:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:06:56.857 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 411 kB/s | 145 kB 00:00 2024-03-29T11:06:56.857 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:56.857 INFO:teuthology.orchestra.run.smithi105.stdout:Total 409 kB/s | 145 kB 00:00 2024-03-29T11:06:56.858 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:06:56.865 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:06:56.865 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:06:56.919 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:06:56.920 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:06:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:06:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:57.131 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:57.131 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:57.132 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:06:57.132 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:57.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:57.207 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:57.207 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:06:57.208 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:57.208 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:57.208 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:57.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-03-29T11:06:58.064 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:58.158 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:58.162 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:06:58.214 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:58.215 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:06:58.215 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:06:58.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:06:58.446 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-03-29T11:06:58.857 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:06:59.007 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:06:59.008 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-29T11:06:59.008 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:06:59.008 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-29T11:06:59.008 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:06:59.008 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:59.009 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.9 M 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-03-29T11:06:59.010 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:06:59.035 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:59.134 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:06:59.187 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:59.188 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:06:59.188 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:06:59.253 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:06:59.462 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 80 kB/s | 15 kB 00:00 2024-03-29T11:06:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 130 kB/s | 31 kB 00:00 2024-03-29T11:06:59.688 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-29T11:06:59.713 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-29T11:06:59.811 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:06:59.955 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-29T11:06:59.960 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout:====================================================================================== 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout:====================================================================================== 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:06:59.962 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout:====================================================================================== 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-03-29T11:06:59.963 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:06:59.964 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 6.9 M 2024-03-29T11:06:59.964 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-03-29T11:06:59.964 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:00.197 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-29T11:07:00.344 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 173 kB/s | 31 kB 00:00 2024-03-29T11:07:00.364 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 842 kB/s | 548 kB 00:00 2024-03-29T11:07:00.406 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.6 MB/s | 1.8 MB 00:01 2024-03-29T11:07:00.445 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 54 kB/s | 15 kB 00:00 2024-03-29T11:07:00.464 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-29T11:07:00.503 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:07:00.528 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T11:07:00.579 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.3 MB/s | 1.8 MB 00:00 2024-03-29T11:07:00.656 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-03-29T11:07:00.729 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-03-29T11:07:00.771 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 115 kB/s | 19 kB 00:00 2024-03-29T11:07:00.855 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:07:00.880 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:07:00.915 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 419 kB/s | 188 kB 00:00 2024-03-29T11:07:00.930 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-03-29T11:07:00.955 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-03-29T11:07:00.972 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-29T11:07:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Total 6.8 MB/s | 6.9 MB 00:01 2024-03-29T11:07:00.995 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:01.012 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:01.012 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:01.048 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:07:01.052 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:01.052 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.4 MB/s | 6.9 MB 00:02 2024-03-29T11:07:01.072 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:01.091 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:01.091 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:01.178 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:01.178 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:01.273 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:01.273 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:07:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:07:01.803 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:07:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:07:01.890 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:07:01.934 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:07:01.980 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:07:01.981 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:07:02.020 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:07:02.110 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:07:02.117 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:07:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:07:02.246 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:07:02.266 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:07:02.294 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:07:02.339 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:07:02.383 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:07:02.411 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:02.443 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:07:02.565 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:07:02.608 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:02.686 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:07:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:03.401 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:03.401 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:07:03.402 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:07:03.505 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:03.506 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:07:03.506 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:07:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:07:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:03.731 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:03.840 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:03.841 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:03.981 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:07:04.103 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:07:04.542 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:04.658 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:04.695 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:07:04.696 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-03-29T11:07:04.697 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:04.698 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 37 M 2024-03-29T11:07:04.698 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 209 M 2024-03-29T11:07:04.698 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:04.810 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:04.811 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-29T11:07:04.811 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:07:04.811 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-29T11:07:04.811 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:07:04.812 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:04.813 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 37 M 2024-03-29T11:07:04.814 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 209 M 2024-03-29T11:07:04.814 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:05.021 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-03-29T11:07:05.079 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 298 kB/s | 17 kB 00:00 2024-03-29T11:07:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-29T11:07:05.239 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-29T11:07:05.264 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 702 kB/s | 17 kB 00:00 2024-03-29T11:07:05.288 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-29T11:07:05.298 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:07:05.348 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:07:05.422 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T11:07:05.480 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 217 kB 00:00 2024-03-29T11:07:05.557 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:07:05.606 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-29T11:07:05.615 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:07:05.640 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T11:07:05.741 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-03-29T11:07:05.816 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-03-29T11:07:05.931 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-29T11:07:06.016 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 6.5 MB/s | 7.4 MB 00:01 2024-03-29T11:07:06.033 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T11:07:06.107 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-29T11:07:06.233 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.5 MB/s | 7.4 MB 00:01 2024-03-29T11:07:06.275 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 538 kB/s | 245 kB 00:00 2024-03-29T11:07:06.278 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:06.278 INFO:teuthology.orchestra.run.smithi110.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-29T11:07:06.318 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:06.342 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:06.342 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:06.432 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 390 kB/s | 195 kB 00:00 2024-03-29T11:07:06.457 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 946 kB/s | 803 kB 00:00 2024-03-29T11:07:06.459 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:06.459 INFO:teuthology.orchestra.run.smithi105.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T11:07:06.499 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:06.522 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:06.522 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:06.712 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:06.805 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:06.883 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:06.884 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:07.413 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:07.490 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:07:07.530 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:07:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:07.624 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:07.638 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:07:07.658 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:07:07.669 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:07:07.761 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:07.798 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:07:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:07.865 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:07.942 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:07:07.974 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:08.005 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:07:08.611 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:07:08.712 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:07:08.729 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:07:08.825 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:07:09.985 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:07:10.043 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:07:11.000 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:07:11.024 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:11.072 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:07:11.097 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:07:13.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:07:13.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:07:13.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:07:14.014 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:07:14.015 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:14.015 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:14.015 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:07:14.015 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:14.015 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:07:14.046 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:14.047 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:14.390 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:07:14.419 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:07:14.960 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:14.989 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:15.111 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:15.112 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:15.112 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:07:15.113 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2024-03-29T11:07:15.114 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:15.115 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.0 M 2024-03-29T11:07:15.115 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-03-29T11:07:15.115 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:15.139 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:15.140 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:15.140 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:07:15.141 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-03-29T11:07:15.142 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:15.143 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-03-29T11:07:15.143 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-03-29T11:07:15.143 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:15.310 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 211 kB/s | 24 kB 00:00 2024-03-29T11:07:15.321 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2024-03-29T11:07:15.335 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-03-29T11:07:15.347 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 24 kB 00:00 2024-03-29T11:07:15.386 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:07:15.397 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:07:15.402 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-29T11:07:15.423 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:07:15.428 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-29T11:07:15.448 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:07:15.453 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:07:15.465 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:07:15.482 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 181 kB/s | 52 kB 00:00 2024-03-29T11:07:15.528 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 156 kB/s | 52 kB 00:00 2024-03-29T11:07:15.562 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-29T11:07:15.587 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-29T11:07:15.690 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-29T11:07:15.696 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-29T11:07:15.716 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T11:07:15.746 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 931 kB/s | 203 kB 00:00 2024-03-29T11:07:15.780 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-29T11:07:15.838 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-29T11:07:15.840 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:15.840 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-29T11:07:15.840 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:15.853 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:15.853 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:15.891 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-29T11:07:15.916 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T11:07:15.941 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:15.942 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:16.067 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-29T11:07:16.225 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:16.241 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:16.241 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:16.332 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:16.332 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:16.470 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:16.641 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:07:16.697 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:07:16.752 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:07:16.782 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:07:16.823 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:07:16.843 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:16.916 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:07:17.003 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:07:17.018 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:07:17.067 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:07:17.075 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:07:17.122 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:07:17.152 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:07:17.193 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:07:17.286 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:07:17.387 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:07:17.445 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:07:17.789 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:07:17.839 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:07:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:07:17.921 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:07:17.945 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:18.167 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:07:18.209 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:07:18.243 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:07:18.291 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:07:18.312 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:07:18.714 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:07:18.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:07:18.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:07:18.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:07:18.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:07:18.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:07:19.059 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:19.060 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:19.071 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:07:19.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:07:19.314 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:07:19.429 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:07:19.430 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:07:19.430 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:07:19.430 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:19.430 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:19.682 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:07:19.884 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:20.037 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:20.037 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-29T11:07:20.038 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-03-29T11:07:20.039 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:20.039 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.8 M 2024-03-29T11:07:20.039 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-03-29T11:07:20.039 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:20.242 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:20.392 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:20.392 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-03-29T11:07:20.393 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:20.394 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 6.8 M 2024-03-29T11:07:20.394 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 33 M 2024-03-29T11:07:20.394 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:20.501 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:07:20.526 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 859 kB/s | 248 kB 00:00 2024-03-29T11:07:20.618 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 362 kB/s | 138 kB 00:00 2024-03-29T11:07:20.877 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:07:20.903 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 444 kB/s | 138 kB 00:00 2024-03-29T11:07:20.961 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 672 kB/s | 248 kB 00:00 2024-03-29T11:07:21.060 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 108 kB/s | 58 kB 00:00 2024-03-29T11:07:21.127 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 110 kB/s | 51 kB 00:00 2024-03-29T11:07:21.311 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 677 kB/s | 548 kB 00:00 2024-03-29T11:07:21.313 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:21.314 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-29T11:07:21.327 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:21.338 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:21.338 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:21.353 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2024-03-29T11:07:21.378 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-03-29T11:07:21.427 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:21.428 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:21.579 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 782 kB/s | 548 kB 00:00 2024-03-29T11:07:21.584 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:21.584 INFO:teuthology.orchestra.run.smithi105.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-29T11:07:21.597 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:21.607 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:21.607 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:21.697 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:21.697 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:21.727 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:21.910 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:07:21.962 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:07:21.997 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:22.162 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:07:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:07:22.493 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:07:22.573 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:07:22.635 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:07:22.658 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:07:22.799 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:07:22.861 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:07:22.884 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:07:23.446 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:07:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:07:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:07:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:07:23.707 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:07:23.707 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:23.738 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:24.002 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:07:24.002 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:24.002 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:24.002 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:24.003 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:24.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-03-29T11:07:24.262 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-03-29T11:07:24.565 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:24.714 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:07:24.715 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 964 k 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.8 M 2024-03-29T11:07:24.716 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:24.821 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:24.880 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 79 kB 00:00 2024-03-29T11:07:24.971 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:24.971 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:07:24.971 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:24.971 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:24.972 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 964 k 2024-03-29T11:07:24.973 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.8 M 2024-03-29T11:07:24.973 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:25.188 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 885 kB 00:00 2024-03-29T11:07:25.189 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:25.190 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-29T11:07:25.190 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:25.209 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:25.209 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:25.238 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 296 kB/s | 79 kB 00:00 2024-03-29T11:07:25.282 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:25.283 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:25.459 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:25.598 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:07:25.671 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:07:26.361 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:07:26.361 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:26.691 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:26.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-03-29T11:07:27.238 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 391 kB/s | 885 kB 00:02 2024-03-29T11:07:27.240 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:27.240 INFO:teuthology.orchestra.run.smithi105.stdout:Total 425 kB/s | 964 kB 00:02 2024-03-29T11:07:27.241 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:27.263 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:27.263 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:27.341 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:27.341 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:27.532 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:27.562 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:27.669 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:07:27.719 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 332 k 2024-03-29T11:07:27.720 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.5 M 2024-03-29T11:07:27.721 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:27.741 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:07:27.843 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 561 kB/s | 69 kB 00:00 2024-03-29T11:07:28.052 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 796 kB/s | 264 kB 00:00 2024-03-29T11:07:28.053 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:28.053 INFO:teuthology.orchestra.run.smithi110.stdout:Total 996 kB/s | 332 kB 00:00 2024-03-29T11:07:28.054 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:28.062 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:28.062 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:28.098 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:28.098 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:28.243 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:28.421 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:07:28.424 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:07:28.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:07:28.463 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:07:28.480 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:07:28.480 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:28.480 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:07:28.480 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:28.718 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:29.001 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-volume 2024-03-29T11:07:29.228 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:07:29.543 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:29.544 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:29.603 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:29.763 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:29.764 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:29.764 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-29T11:07:29.764 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:29.764 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 332 k 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 1.5 M 2024-03-29T11:07:29.765 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:29.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-03-29T11:07:29.887 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 566 kB/s | 69 kB 00:00 2024-03-29T11:07:30.104 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 779 kB/s | 264 kB 00:00 2024-03-29T11:07:30.105 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:30.106 INFO:teuthology.orchestra.run.smithi105.stdout:Total 975 kB/s | 332 kB 00:00 2024-03-29T11:07:30.106 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:30.115 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:30.115 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:30.152 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:30.153 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:30.302 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:30.460 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:30.468 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:07:30.506 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:07:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:07:30.523 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:30.523 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:07:30.523 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:30.610 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:30.611 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:30.611 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-29T11:07:30.611 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:30.611 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:30.611 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 127 k 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 456 k 2024-03-29T11:07:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:30.967 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 356 kB/s | 127 kB 00:00 2024-03-29T11:07:30.968 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:30.968 INFO:teuthology.orchestra.run.smithi110.stdout:Total 355 kB/s | 127 kB 00:00 2024-03-29T11:07:30.968 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:30.973 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:30.973 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:31.029 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:31.029 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:31.283 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:07:31.293 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:31.627 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:31.945 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-03-29T11:07:32.045 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:32.045 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:32.045 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:32.046 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:32.046 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:32.046 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:32.303 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-03-29T11:07:32.524 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:32.673 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:32.674 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:32.674 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 127 k 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 456 k 2024-03-29T11:07:32.675 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:32.875 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:32.901 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 561 kB/s | 127 kB 00:00 2024-03-29T11:07:32.902 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:32.902 INFO:teuthology.orchestra.run.smithi105.stdout:Total 557 kB/s | 127 kB 00:00 2024-03-29T11:07:32.902 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:32.907 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:32.908 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:32.962 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:32.963 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:32.976 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:33.031 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:33.032 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:33.032 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:33.108 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-03-29T11:07:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:33.670 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:33.822 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:07:33.824 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 31 k 2024-03-29T11:07:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 140 k 2024-03-29T11:07:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:33.980 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 200 kB/s | 31 kB 00:00 2024-03-29T11:07:33.980 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:33.980 INFO:teuthology.orchestra.run.smithi110.stdout:Total 198 kB/s | 31 kB 00:00 2024-03-29T11:07:33.981 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:33.984 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:33.984 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:34.014 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:34.014 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:34.034 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:34.035 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:34.035 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:34.035 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:34.035 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:34.035 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:34.142 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:34.236 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:34.289 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-03-29T11:07:34.626 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:34.863 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:34.877 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:34.964 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:35.017 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:35.018 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:35.018 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:35.036 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-03-29T11:07:35.080 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-03-29T11:07:35.597 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:35.649 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:35.693 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:35.744 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:35.745 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:35.745 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:35.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-03-29T11:07:35.802 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:35.802 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-29T11:07:35.802 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:35.802 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:07:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:35.804 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 31 k 2024-03-29T11:07:35.804 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 140 k 2024-03-29T11:07:35.804 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:36.046 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-29T11:07:36.047 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:36.047 INFO:teuthology.orchestra.run.smithi105.stdout:Total 128 kB/s | 31 kB 00:00 2024-03-29T11:07:36.047 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:36.050 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:36.050 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:36.074 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:36.074 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:36.203 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:36.310 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:36.361 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:36.459 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:36.511 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:36.512 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:36.512 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:36.564 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-03-29T11:07:36.703 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:36.973 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:37.126 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-03-29T11:07:37.129 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:37.228 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:37.280 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:37.280 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:37.281 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:37.342 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-03-29T11:07:37.685 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:37.832 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:37.833 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:37.834 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:37.885 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-03-29T11:07:37.912 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:38.011 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:38.062 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:38.063 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:38.063 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:38.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-03-29T11:07:38.446 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:38.542 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:38.593 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:38.594 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:38.595 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:38.643 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-03-29T11:07:38.682 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:38.779 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:38.831 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:38.832 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:38.832 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:38.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-03-29T11:07:39.207 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:39.305 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:39.357 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:39.358 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:39.358 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:39.412 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-03-29T11:07:39.446 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:39.544 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:39.597 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:39.598 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-29T11:07:39.598 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:39.647 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-03-29T11:07:39.983 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:40.080 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:40.132 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:40.133 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:40.133 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:40.186 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-03-29T11:07:40.209 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:40.358 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:07:40.359 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:40.360 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 85 k 2024-03-29T11:07:40.360 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 231 k 2024-03-29T11:07:40.360 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:40.724 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 234 kB/s | 85 kB 00:00 2024-03-29T11:07:40.725 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:40.725 INFO:teuthology.orchestra.run.smithi110.stdout:Total 233 kB/s | 85 kB 00:00 2024-03-29T11:07:40.725 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:40.732 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:40.733 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:40.759 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:40.787 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:40.788 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:40.857 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:40.909 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:40.910 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:40.910 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:40.929 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:40.964 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-03-29T11:07:41.038 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:41.401 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:41.523 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:41.621 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:41.672 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:41.673 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-29T11:07:41.673 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:41.685 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:41.685 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:41.686 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:41.686 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:41.686 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:41.686 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:41.727 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-03-29T11:07:41.904 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-03-29T11:07:42.289 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:42.437 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:42.438 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:42.438 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-03-29T11:07:42.438 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 85 k 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 231 k 2024-03-29T11:07:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:42.489 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:42.642 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:07:42.643 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.1 M 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 13 M 2024-03-29T11:07:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:42.783 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 248 kB/s | 85 kB 00:00 2024-03-29T11:07:42.783 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:42.784 INFO:teuthology.orchestra.run.smithi105.stdout:Total 247 kB/s | 85 kB 00:00 2024-03-29T11:07:42.784 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:42.791 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:42.791 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:42.849 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:42.849 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:43.098 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:43.502 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:43.791 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:43.792 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:44.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-03-29T11:07:44.404 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:07:44.405 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:44.405 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:07:44.405 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:44.415 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:44.415 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:44.489 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:44.489 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:44.592 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:44.747 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:44.748 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:44.748 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-29T11:07:44.748 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:44.748 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.1 M 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 13 M 2024-03-29T11:07:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:44.924 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:44.988 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:07:45.013 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:45.227 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:07:45.228 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:45.228 INFO:teuthology.orchestra.run.smithi105.stdout:Total 6.4 MB/s | 3.1 MB 00:00 2024-03-29T11:07:45.228 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:45.237 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:45.238 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:45.311 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:45.311 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:45.736 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:45.786 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:45.810 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:45.810 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:45.810 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:07:45.811 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:07:45.811 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:07:45.811 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:46.145 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:46.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-03-29T11:07:46.959 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:05:57 AM UTC. 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:46.964 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:47.106 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:47.107 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 171 k 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 494 k 2024-03-29T11:07:47.108 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-29T11:07:47.214 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-03-29T11:07:47.458 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 490 kB/s | 171 kB 00:00 2024-03-29T11:07:47.459 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:47.459 INFO:teuthology.orchestra.run.smithi110.stdout:Total 488 kB/s | 171 kB 00:00 2024-03-29T11:07:47.459 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-29T11:07:47.467 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-29T11:07:47.467 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-29T11:07:47.524 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-29T11:07:47.525 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-29T11:07:47.668 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-29T11:07:47.781 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:47.787 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:06:00 AM UTC. 2024-03-29T11:07:47.940 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:07:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 171 k 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 494 k 2024-03-29T11:07:47.942 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-29T11:07:48.185 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:48.250 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 563 kB/s | 171 kB 00:00 2024-03-29T11:07:48.250 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:48.250 INFO:teuthology.orchestra.run.smithi105.stdout:Total 560 kB/s | 171 kB 00:00 2024-03-29T11:07:48.250 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-29T11:07:48.257 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-29T11:07:48.257 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-29T11:07:48.311 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-29T11:07:48.311 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-29T11:07:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:48.466 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-29T11:07:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:48.720 DEBUG:teuthology.parallel:result is None 2024-03-29T11:07:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:49.289 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-29T11:07:49.520 DEBUG:teuthology.parallel:result is None 2024-03-29T11:07:49.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:49.658 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:07:49.679 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:07:49.679 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:07:49.680 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:07:49.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:49.815 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:07:49.840 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:07:49.840 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:07:49.840 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:07:49.842 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:07:49.842 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:49.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:07:49.870 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:49.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:07:49.911 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:07:49.911 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:49.911 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:07:49.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:07:50.004 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:50.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:07:50.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:07:50.104 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:07:50.105 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:50.105 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:07:50.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:07:50.200 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:50.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:07:50.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:07:50.299 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:07:50.300 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:50.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:07:50.327 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:07:50.392 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:50.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:07:50.423 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:07:50.492 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:07:50.573 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:07:50.573 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2024-03-29T11:07:50.575 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-03-29T11:07:50.607 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:07:50.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:07:50.643 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:07:50.676 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:07:50.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:07:50.709 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:07:50.756 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:07:50.756 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:07:50.756 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622675', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi105.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGe2cbzlf1n6FisHkjzCBQV8qL7Y7EdolAUPvoVYE70dLHfL29aFbDMXR0dueLZBqTXP40Xs8fCa9gEC2xFZUw=', 'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2KushoYANTljforGMe/c4ZdrBe6UZpibQLuMPDjQiS3b4uSst/CWLgNi/sc8NskVETh48c5oBQyYnNV/cicqo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:07:50.756 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:07:50.776 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:07:50.802 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:07:50.830 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:07:50.870 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:50.871 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:07:50.910 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:07:50.910 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-29 11:07:46.091657850 +0000 2024-03-29T11:07:50.965 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-29 11:05:09.742545945 +0000 2024-03-29T11:07:50.966 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-29 11:05:09.742545945 +0000 2024-03-29T11:07:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-29 11:05:09.742545945 +0000 2024-03-29T11:07:50.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:07:51.030 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-29T11:07:51.030 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-29T11:07:51.030 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000177771 s, 2.9 MB/s 2024-03-29T11:07:51.032 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:07:51.087 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-03-29T11:07:51.141 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-29 11:07:46.091657850 +0000 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-29 11:05:10.096538684 +0000 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-29 11:05:10.096538684 +0000 2024-03-29T11:07:51.142 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-29 11:05:10.096538684 +0000 2024-03-29T11:07:51.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:07:51.207 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-29T11:07:51.207 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-29T11:07:51.207 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000247635 s, 2.1 MB/s 2024-03-29T11:07:51.208 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:07:51.264 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:51.318 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-29 11:07:46.092657830 +0000 2024-03-29T11:07:51.319 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-29 11:05:10.444531546 +0000 2024-03-29T11:07:51.319 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-29 11:05:10.444531546 +0000 2024-03-29T11:07:51.319 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-29 11:05:10.444531546 +0000 2024-03-29T11:07:51.319 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:07:51.384 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-29T11:07:51.384 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-29T11:07:51.384 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000187114 s, 2.7 MB/s 2024-03-29T11:07:51.385 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:07:51.441 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-03-29T11:07:51.496 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:07:51.496 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:51.496 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:07:51.496 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:51.497 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:51.497 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-29 11:07:46.092657830 +0000 2024-03-29T11:07:51.497 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-29 11:05:10.792524410 +0000 2024-03-29T11:07:51.497 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-29 11:05:10.792524410 +0000 2024-03-29T11:07:51.497 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-29 11:05:10.792524410 +0000 2024-03-29T11:07:51.497 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:07:51.561 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-29T11:07:51.561 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-29T11:07:51.561 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000188387 s, 2.7 MB/s 2024-03-29T11:07:51.562 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:07:51.618 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:51.618 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:51.618 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:07:51.636 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:07:51.636 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-29 11:07:45.274450803 +0000 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-29 11:05:11.110868063 +0000 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-29 11:05:11.110868063 +0000 2024-03-29T11:07:51.694 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-29 11:05:11.110868063 +0000 2024-03-29T11:07:51.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:07:51.762 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-29T11:07:51.762 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-29T11:07:51.762 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000118406 s, 4.3 MB/s 2024-03-29T11:07:51.763 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:07:51.822 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-29 11:07:45.275450785 +0000 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-29 11:05:11.438862048 +0000 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-29 11:05:11.438862048 +0000 2024-03-29T11:07:51.879 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-29 11:05:11.438862048 +0000 2024-03-29T11:07:51.880 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:07:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-29T11:07:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-29T11:07:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000109586 s, 4.7 MB/s 2024-03-29T11:07:51.948 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:07:52.005 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-29 11:07:45.275450785 +0000 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-29 11:05:11.747856381 +0000 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-29 11:05:11.747856381 +0000 2024-03-29T11:07:52.061 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-29 11:05:11.747856381 +0000 2024-03-29T11:07:52.062 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:07:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-29T11:07:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-29T11:07:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000109477 s, 4.7 MB/s 2024-03-29T11:07:52.130 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:07:52.186 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-03-29T11:07:52.240 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:07:52.240 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:07:52.240 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-29 11:07:45.275450785 +0000 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-29 11:05:12.060850641 +0000 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-29 11:05:12.060850641 +0000 2024-03-29T11:07:52.241 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-29 11:05:12.060850641 +0000 2024-03-29T11:07:52.241 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:07:52.306 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-29T11:07:52.306 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-29T11:07:52.306 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.7643e-05 s, 6.6 MB/s 2024-03-29T11:07:52.307 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:07:52.362 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:52.362 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:07:52.362 INFO:tasks.ceph:Generating config... 2024-03-29T11:07:52.363 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:07:52.363 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:07:52.363 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:07:52.364 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:07:52.364 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:07:52.364 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:07:52.365 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:07:52.365 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:07:52.365 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:07:52.365 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:07:52.365 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:07:52.365 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:07:52.366 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:07:52.367 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:07:52.367 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:07:52.367 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:07:52.421 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:07:52.424 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:07:52.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:07:52.495 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.105:6789]'), ('mon.c', '[v1:172.21.15.105:6790]'), ('mon.b', '[v1:172.21.15.110:6789]')] 2024-03-29T11:07:52.495 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.105:6789],[v1:172.21.15.105:6790],[v1:172.21.15.110:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:07:52.496 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:52.497 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:07:52.550 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.105:6789]' --addv c '[v1:172.21.15.105:6790]' --addv b '[v1:172.21.15.110:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:52.634 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: generated fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = quincy 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 0 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:min_mon_release 17 (quincy) 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:election_strategy: 1 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:07:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:07:52.637 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:07:52.651 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 39787f52-b512-4aae-b655-f9112d1c907c... 2024-03-29T11:07:52.652 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:07:52.693 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:07:52.741 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-03-29T11:07:52.741 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2024-03-29T11:07:52.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2024-03-29T11:07:52.743 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = "[v1:172.21.15.105:6789],[v1:172.21.15.105:6790],[v1:172.21.15.110:6789]" 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: lockdep = True 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: mon election default strategy = 1 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: ms bind msgr2 = False 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 5000 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: ms type = async 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 600 2024-03-29T11:07:52.745 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug trim objects = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.746 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression algorithm = zstd 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression mode = aggressive 2024-03-29T11:07:52.747 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore zero block detection = True 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 20 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 20 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug monc = 20 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 10 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: osd class default list = * 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: osd class load list = * 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: osd client watch timeout = 120 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify cached snaps = True 2024-03-29T11:07:52.748 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify missing on start = True 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock override recovery settings = True 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue = debug_random 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue cut off = debug_random 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: osd shutdown pgref assert = True 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:07:52.749 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2024-03-29T11:07:52.750 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: # 1m isn't quite enough 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:07:52.751 INFO:teuthology.orchestra.run.smithi105.stdout: mon min osdmap epochs = 50 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune interval = 2 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune min = 15 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: mon scrub interval = 300 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on pool no app = False 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: paxos service trim min = 10 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: debug monc = 20 2024-03-29T11:07:52.752 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi105.stdout: debug objecter = 20 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi105.stdout: debug rados = 20 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.c] 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.b] 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:07:52.753 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-03-29T11:07:52.754 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:07:52.755 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "[v1:172.21.15.105:6789],[v1:172.21.15.105:6790],[v1:172.21.15.110:6789]" 2024-03-29T11:07:52.756 INFO:teuthology.orchestra.run.smithi110.stdout: lockdep = True 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr2 = False 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 5000 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.757 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-03-29T11:07:52.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore compression algorithm = zstd 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore compression mode = aggressive 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 20 2024-03-29T11:07:52.759 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 20 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd class default list = * 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd class load list = * 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd client watch timeout = 120 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:07:52.760 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-03-29T11:07:52.761 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-03-29T11:07:52.763 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on pool no app = False 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: debug objecter = 20 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout: debug rados = 20 2024-03-29T11:07:52.764 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-03-29T11:07:52.765 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-03-29T11:07:52.765 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-03-29T11:07:52.766 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:07:52.766 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:07:52.862 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:07:52.862 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:52.862 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:07:52.877 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:52.877 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:07:52.932 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:07:52.932 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:52.932 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:07:52.933 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:07:53.010 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:07:53.010 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:53.065 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:07:53.066 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:53.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:07:53.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:07:53.105 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:07:53.105 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:53.160 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:07:53.161 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:07:53.161 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:07:53.215 INFO:teuthology.orchestra.run.smithi105.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:07:53.217 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:07:53.275 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:07:53.277 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:07:53.277 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:07:53.278 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:07:53.320 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:07:53.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:07:53.377 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:07:53.392 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:07:53.392 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:07:53.393 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:07:53.419 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:53.419 INFO:tasks.ceph:role: osd.0 2024-03-29T11:07:53.419 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:07:53.419 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:07:53.493 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:07:53.493 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:07:53.494 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:07:53.988 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-03-29T11:07:53.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-03-29T11:07:53.990 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:07:54.135 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:07:54.167 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:07:54.234 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:54.234 INFO:tasks.ceph:role: osd.1 2024-03-29T11:07:54.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:07:54.234 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:07:54.309 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:07:54.765 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-03-29T11:07:54.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-03-29T11:07:54.767 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:07:54.862 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:07:54.893 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:07:54.960 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:54.960 INFO:tasks.ceph:role: osd.2 2024-03-29T11:07:54.960 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:07:54.960 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:07:55.034 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:07:55.034 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:07:55.035 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:07:55.523 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-03-29T11:07:55.525 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-03-29T11:07:55.525 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:07:55.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:07:55.710 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:07:55.777 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:07:55.777 INFO:tasks.ceph:role: osd.3 2024-03-29T11:07:55.777 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:07:55.777 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:07:55.852 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:07:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:07:55.853 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:07:56.356 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-03-29T11:07:56.357 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-03-29T11:07:56.358 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:07:56.486 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:07:56.521 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.645+0000 7f63a4b93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.645+0000 7f63a4b93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.645+0000 7f63a4b93640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.645+0000 7f63a4b93640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.645+0000 7f63a4b93640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:07:56.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:56.646+0000 7f63a4b93640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:07:59.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:07:59.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:07:59.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.628+0000 7fcd49306640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:07:59.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.628+0000 7fcd49306640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:07:59.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.628+0000 7fcd49306640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:07:59.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.628+0000 7fcd49306640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:07:59.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.628+0000 7fcd49306640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:07:59.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:07:59.629+0000 7fcd49306640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:08:02.451 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:02.482 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.595+0000 7f9212ed9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.595+0000 7f9212ed9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.595+0000 7f9212ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.595+0000 7f9212ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.595+0000 7f9212ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:02.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:02.596+0000 7f9212ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:08:05.426 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:05.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.568+0000 7f2a350b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:08:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.568+0000 7f2a350b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:08:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.568+0000 7f2a350b5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.568+0000 7f2a350b5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.568+0000 7f2a350b5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:05.569+0000 7f2a350b5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:08:08.406 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:08.437 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:08.469 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:08.469 INFO:tasks.ceph:role: osd.4 2024-03-29T11:08:08.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:08:08.469 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:08.539 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:08.540 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:08.540 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:08.571 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-29T11:08:08.573 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-29T11:08:08.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:08.647 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:08.718 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:08.786 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:08.787 INFO:tasks.ceph:role: osd.5 2024-03-29T11:08:08.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:08:08.787 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:08:08.853 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:08.853 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:08.854 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:08.885 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-29T11:08:08.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-29T11:08:08.989 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:09.070 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:09.155 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:09.231 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:09.231 INFO:tasks.ceph:role: osd.6 2024-03-29T11:08:09.231 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:08:09.231 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:09.332 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-29T11:08:09.334 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-29T11:08:09.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:09.410 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:09.482 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:09.550 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:09.550 INFO:tasks.ceph:role: osd.7 2024-03-29T11:08:09.550 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:08:09.550 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:08:09.618 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:09.618 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:09.619 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:09.619 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:09.619 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:09.651 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-29T11:08:09.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-29T11:08:09.653 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:09.724 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:09.798 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:09.916+0000 7f8b34d73640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:08:12.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:12.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:12.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.889+0000 7f0e00c15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:08:12.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.889+0000 7f0e00c15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:08:12.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.889+0000 7f0e00c15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:12.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.889+0000 7f0e00c15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:12.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.889+0000 7f0e00c15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:12.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:12.890+0000 7f0e00c15640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:08:15.718 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:15.748 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:15.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.860+0000 7f3123af5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:08:15.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.861+0000 7f3123af5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:08:15.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.861+0000 7f3123af5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:15.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.861+0000 7f3123af5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:15.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.861+0000 7f3123af5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:15.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:15.861+0000 7f3123af5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:08:18.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:18.712 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:18.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.820+0000 7f7566cb3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:08:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.820+0000 7f7566cb3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:08:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.820+0000 7f7566cb3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.820+0000 7f7566cb3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.820+0000 7f7566cb3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:18.821+0000 7f7566cb3640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:08:21.650 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:21.681 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:08:21.681 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:21.681 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:08:21.714 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:21.714 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:08:21.783 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:21.784 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:08:21.852 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:21.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:08:21.921 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:21.921 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:08:21.988 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:21.988 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:08:22.020 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:22.020 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:08:22.090 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:22.090 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:08:22.158 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:22.158 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:08:22.226 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:22.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:08:22.293 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:22.293 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:08:22.310 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:22.310 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:08:22.348 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:08:22.348 DEBUG:teuthology.orchestra.run.smithi105:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:08:22.353 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.y] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAJoQZmhPcqDhAAxieSnrs/sS8j34EK0iuSkw== 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.0] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAMoQZmg2CFJhAAIG4eLXtZBCb5yUy5oz8ZVA== 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.1] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAPoQZm9L6EJRAAvfwxwIQbFkmQRUtPlhOtlQ== 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.2] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQASoQZmpOyIIxAA6nQLrCd6fmVjP7nu0NZ1fQ== 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.3] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAVoQZmoVjzIRAACVwRtLKsuZYQgNdvOeRTeQ== 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.x] 2024-03-29T11:08:22.391 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAJoQZmtCG/ERAAquYIgjAbF8g5rk5hCxjIPA== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.4] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAZoQZmVTueNhAAkSaeoTf1IgRu4AT+TSeSiA== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.5] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAcoQZmBuoXNRAAmGGRwHQhq9ZibI8/9sdfpQ== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.6] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAfoQZmSBhjMxAAoPfP4HzhEc8VMvbXi/Y0vQ== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.7] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAioQZmSiz+MBAAXkv0mnhWsr4LnQhKxp3UGw== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAJoQZmCoJwFBAAGEHI8VeaEIrGHEPbjr7GHg== 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout:[client.1] 2024-03-29T11:08:22.392 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAJoQZmcmvUFxAAHgoFCYoM7aUgYiGw58jmZQ== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAJoQZmhPcqDhAAxieSnrs/sS8j34EK0iuSkw== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAMoQZmg2CFJhAAIG4eLXtZBCb5yUy5oz8ZVA== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAPoQZm9L6EJRAAvfwxwIQbFkmQRUtPlhOtlQ== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQASoQZmpOyIIxAA6nQLrCd6fmVjP7nu0NZ1fQ== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAVoQZmoVjzIRAACVwRtLKsuZYQgNdvOeRTeQ== 2024-03-29T11:08:22.413 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAJoQZmtCG/ERAAquYIgjAbF8g5rk5hCxjIPA== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAZoQZmVTueNhAAkSaeoTf1IgRu4AT+TSeSiA== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAcoQZmBuoXNRAAmGGRwHQhq9ZibI8/9sdfpQ== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAfoQZmSBhjMxAAoPfP4HzhEc8VMvbXi/Y0vQ== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAioQZmSiz+MBAAXkv0mnhWsr4LnQhKxp3UGw== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAJoQZmCoJwFBAAGEHI8VeaEIrGHEPbjr7GHg== 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-03-29T11:08:22.414 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAJoQZmcmvUFxAAHgoFCYoM7aUgYiGw58jmZQ== 2024-03-29T11:08:22.415 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:22.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:22.507 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.509 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.561 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.563 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.614 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.616 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.670 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:22.723 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:22.774 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.827 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.829 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.880 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.934 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.936 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:22.987 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:22.989 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:23.044 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:23.046 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:23.101 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:08:23.101 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:08:23.130 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:23.227 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:08:23.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:08:23.320 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:23.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:08:23.443 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:08:23.470 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:23.568 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:08:23.595 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:23.597 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:23.649 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:08:23.649 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:08:23.650 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:08:23.652 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:08:23.652 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:08:23.652 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:08:23.654 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:08:23.654 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:08:23.654 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:08:23.692 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:08:23.692 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:08:23.692 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:08:23.692 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:08:23.694 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:08:23.694 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:08:23.694 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:08:23.696 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:08:23.697 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:08:23.697 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.818+0000 7fe5525cf640 10 monclient: get_monmap_and_config 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.818+0000 7fe5525cf640 10 monclient: build_initial_monmap 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.819+0000 7fe5525cf640 10 monclient: monmap: 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 0 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:08:23.820022+0000 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:08:23.820022+0000 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:23.798 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.819+0000 7fe5525cf640 1 Processor -- start 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.819+0000 7fe5525cf640 1 -- start start 2024-03-29T11:08:23.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.819+0000 7fe5525cf640 10 monclient: init 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c09ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c09c870 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe54c0b4360 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c09dec0 con 0x7fe54c09c870 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c09da30 con 0x7fe54c0b4360 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0b39c0 con 0x7fe54c09ac40 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe5525cf640 10 monclient(hunting): authenticate will time out at 778.096420s 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe54bfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fe54c09ac40 0x7fe54c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:39372/0 (socket says 172.21.15.105:39372) 2024-03-29T11:08:23.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe54b7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fe54c09c870 0x7fe54c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:50754/0 (socket says 172.21.15.105:50754) 2024-03-29T11:08:23.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:23.820+0000 7fe54b7fe640 1 -- 172.21.15.105:0/3734958197 learned_addr learned my addr 172.21.15.105:0/3734958197 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:26.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): tick 2024-03-29T11:08:26.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 monclient(hunting): continuing hunt 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.105:6790/0 conn(0x7fe54c09ac40 legacy=0x7fe54c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.110:6789/0 conn(0x7fe54c0b4360 legacy=0x7fe54c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.105:6789/0 conn(0x7fe54c09c870 legacy=0x7fe54c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5380036e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c09ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe53800ab40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0b5160 con 0x7fe5380036e0 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53800f810 con 0x7fe53800ab40 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe538010a50 con 0x7fe54c09ac40 2024-03-29T11:08:26.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:26.820+0000 7fe549ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:28.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424020968 0 0) 0x7fe54c0b5160 con 0x7fe5380036e0 2024-03-29T11:08:28.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.985+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534003520 con 0x7fe5380036e0 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260275935 0 0) 0x7fe538010a50 con 0x7fe54c09ac40 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54c0b5160 con 0x7fe54c09ac40 2024-03-29T11:08:28.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286151826 0 0) 0x7fe53800f810 con 0x7fe53800ab40 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538010a50 con 0x7fe53800ab40 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3967918144 0 0) 0x7fe54c0b5160 con 0x7fe54c09ac40 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.110:6789/0 conn(0x7fe53800ab40 legacy=0x7fe53800cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.105:6789/0 conn(0x7fe5380036e0 legacy=0x7fe538005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 monclient: found mon.noname-b 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:28.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538011cb0 con 0x7fe54c09ac40 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: _finish_auth 0 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe540001470 con 0x7fe54c09ac40 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe54affd640 10 monclient: dump: 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.968 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe5525cf640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.986+0000 7fe5525cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5400015d0 con 0x7fe54c09ac40 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 1 -- 172.21.15.105:0/3734958197 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe5400019e0 con 0x7fe54c09ac40 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:28.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe54affd640 10 monclient: dump: 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:28.970 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:28.971 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:28.971 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:28.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe5525cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:28.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.987+0000 7fe5525cf640 4 set_mon_vals no callback set 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 10 monclient: shutdown 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 1 -- 172.21.15.105:0/3734958197 >> v1:172.21.15.105:6790/0 conn(0x7fe54c09ac40 legacy=0x7fe538009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 1 -- 172.21.15.105:0/3734958197 shutdown_connections 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 1 -- 172.21.15.105:0/3734958197 >> 172.21.15.105:0/3734958197 conn(0x7fe54c066b00 msgr2=0x7fe54c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 1 -- 172.21.15.105:0/3734958197 shutdown_connections 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.988+0000 7fe5525cf640 1 -- 172.21.15.105:0/3734958197 wait complete. 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.989+0000 7fe5525cf640 10 monclient: build_initial_monmap 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.989+0000 7fe5525cf640 10 monclient: monmap: 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 0 2024-03-29T11:08:28.977 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:created 0.000000 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.989+0000 7fe5525cf640 1 librados: starting msgr at 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.989+0000 7fe5525cf640 1 librados: starting objecter 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.990+0000 7fe5525cf640 1 Processor -- start 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.990+0000 7fe5525cf640 1 -- start start 2024-03-29T11:08:28.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.990+0000 7fe5525cf640 1 librados: setting wanted keys 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.990+0000 7fe5525cf640 1 librados: calling monclient init 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.990+0000 7fe5525cf640 10 monclient: init 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe5380036e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe53800ab40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c09ac40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c1d7130 con 0x7fe54c09ac40 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0b6bb0 con 0x7fe53800ab40 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c1d7d60 con 0x7fe5380036e0 2024-03-29T11:08:28.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe5525cf640 10 monclient(hunting): authenticate will time out at 783.267111s 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe54bfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fe5380036e0 0x7fe54c1d7650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:35004/0 (socket says 172.21.15.105:35004) 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe54bfff640 1 -- 172.21.15.105:0/3783654701 learned_addr learned my addr 172.21.15.105:0/3783654701 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe5380036e0 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe54c09ac40 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe53800ab40 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727986349 0 0) 0x7fe54c0b6bb0 con 0x7fe53800ab40 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.991+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538006b00 con 0x7fe53800ab40 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509418927 0 0) 0x7fe54c1d7d60 con 0x7fe5380036e0 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54c0b6bb0 con 0x7fe5380036e0 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4239786066 0 0) 0x7fe538006b00 con 0x7fe53800ab40 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c1d7d60 con 0x7fe53800ab40 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe540001d80 con 0x7fe53800ab40 2024-03-29T11:08:28.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): dump: 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:28.982 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1391186586 0 0) 0x7fe54c1d7d60 con 0x7fe53800ab40 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 >> v1:172.21.15.105:6790/0 conn(0x7fe5380036e0 legacy=0x7fe54c1d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 >> v1:172.21.15.105:6789/0 conn(0x7fe54c09ac40 legacy=0x7fe54c0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 monclient: found mon.b 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c1d7ef0 con 0x7fe53800ab40 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: _finish_auth 0 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:28.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe5525cf640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe5525cf640 10 monclient: _renew_subs 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe5525cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c144c40 con 0x7fe53800ab40 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe540004a90 con 0x7fe53800ab40 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe540004da0 con 0x7fe53800ab40 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe548ff9640 10 monclient: dump: 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.984 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.992+0000 7fe54a7fc640 4 set_mon_vals no callback set 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe540005290 con 0x7fe53800ab40 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe5525cf640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe5525cf640 10 monclient: _renew_subs 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe5525cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c13b690 con 0x7fe53800ab40 2024-03-29T11:08:28.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe5525cf640 1 librados: init done 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 85886371 0 0) 0x7fe540005a10 con 0x7fe53800ab40 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 10 client.4114.objecter ms_dispatch 0x7fe54c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 3 client.4114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 3 client.4114.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.993+0000 7fe548ff9640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.994+0000 7fe5525cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.994+0000 7fe5525cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.994+0000 7fe5525cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.994+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0acfc0 con 0x7fe53800ab40 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.997+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe540005bd0 con 0x7fe53800ab40 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.997+0000 7fe548ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:28.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:28.997+0000 7fe548ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:29.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.069+0000 7fe5525cf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:08:29.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.069+0000 7fe5525cf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:08:29.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.069+0000 7fe5525cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.069+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe54c09b020 con 0x7fe53800ab40 2024-03-29T11:08:29.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.072+0000 7fe548ff9640 1 -- 172.21.15.105:0/3783654701 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe540005e50 con 0x7fe53800ab40 2024-03-29T11:08:29.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.072+0000 7fe548ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:08:29.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.072+0000 7fe548ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:08:29.051 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 10 librados: watch_flush enter 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 10 librados: watch_flush exit 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 10 monclient: shutdown 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 >> v1:172.21.15.110:6789/0 conn(0x7fe53800ab40 legacy=0x7fe54c13ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 shutdown_connections 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 >> 172.21.15.105:0/3783654701 conn(0x7fe54c066b00 msgr2=0x7fe54c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 shutdown_connections 2024-03-29T11:08:29.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 -- 172.21.15.105:0/3783654701 wait complete. 2024-03-29T11:08:29.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:29.073+0000 7fe5525cf640 1 librados: shutdown 2024-03-29T11:08:29.063 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:08:29.063 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:08:29.064 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:29.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:08:29.091 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:29.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:08:29.158 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:29.158 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:08:29.225 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-29T11:08:29.225 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:08:29.292 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:29.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:08:29.320 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:29.320 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:08:29.387 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:29.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:08:29.450 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:29.471+0000 7f8921e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:08:29.453 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-29T11:08:29.453 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:08:29.465 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:29.486+0000 7fd612032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:08:29.506 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:29.527+0000 7f8921e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:08:29.518 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:29.539+0000 7fd612032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:08:29.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2 0 2024-03-29T11:08:29.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.681+0000 7f80bddc8640 10 monclient: get_monmap_and_config 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.681+0000 7f80bddc8640 10 monclient: build_initial_monmap 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.681+0000 7f80bddc8640 10 monclient: monmap: 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:29.683072+0000 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:29.683072+0000 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:29.661 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.682+0000 7f80bddc8640 1 Processor -- start 2024-03-29T11:08:29.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.682+0000 7f80bddc8640 1 -- start start 2024-03-29T11:08:29.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.682+0000 7f80bddc8640 10 monclient: init 2024-03-29T11:08:29.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:29.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-b con 0x7f80b809ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-c con 0x7f80b809c870 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-a con 0x7f80b80b4360 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b809dec0 con 0x7f80b80b4360 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b809da30 con 0x7f80b809c870 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b80b39c0 con 0x7f80b809ac40 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient(hunting): authenticate will time out at 789.874274s 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b6ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f80b809c870 0x7f80b813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56618/0 (socket says 172.21.15.110:56618) 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b6ffd640 1 -- 172.21.15.110:0/486887876 learned_addr learned my addr 172.21.15.110:0/486887876 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25863804 0 0) 0x7f80b809da30 con 0x7f80b809c870 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a0003520 con 0x7f80b809c870 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038480220 0 0) 0x7f80b0001490 con 0x7f80b80b4360 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b809da30 con 0x7f80b80b4360 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811285926 0 0) 0x7f80b80b39c0 con 0x7f80b809ac40 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b0001490 con 0x7f80b809ac40 2024-03-29T11:08:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453543084 0 0) 0x7f80a0003520 con 0x7f80b809c870 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 >> v1:172.21.15.105:6790/0 conn(0x7f80b809ac40 legacy=0x7f80b80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 >> v1:172.21.15.105:6789/0 conn(0x7f80b80b4360 legacy=0x7f80b814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 monclient: found mon.noname-c 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b80b5160 con 0x7f80b809c870 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80ac001690 con 0x7f80b809c870 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: dump: 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80ac0017f0 con 0x7f80b809c870 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 1 -- 172.21.15.110:0/486887876 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80ac001c00 con 0x7f80b809c870 2024-03-29T11:08:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80bddc8640 4 set_mon_vals no callback set 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.683+0000 7f80b67fc640 10 monclient: dump: 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 10 monclient: shutdown 2024-03-29T11:08:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 1 -- 172.21.15.110:0/486887876 >> v1:172.21.15.110:6789/0 conn(0x7f80b809c870 legacy=0x7f80b813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 1 -- 172.21.15.110:0/486887876 shutdown_connections 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 1 -- 172.21.15.110:0/486887876 >> 172.21.15.110:0/486887876 conn(0x7f80b8066b00 msgr2=0x7f80b8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 1 -- 172.21.15.110:0/486887876 shutdown_connections 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.684+0000 7f80bddc8640 1 -- 172.21.15.110:0/486887876 wait complete. 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 10 monclient: build_initial_monmap 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 10 monclient: monmap: 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 librados: starting msgr at 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 librados: starting objecter 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 Processor -- start 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 -- start start 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 librados: setting wanted keys 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 1 librados: calling monclient init 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.685+0000 7f80bddc8640 10 monclient: init 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-b con 0x7f80b809ac40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-c con 0x7f80b80acd00 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient(hunting): picked mon.noname-a con 0x7f80b80b0320 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b81da5a0 con 0x7f80b80b0320 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b80b1520 con 0x7f80b809ac40 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b80af130 con 0x7f80b80acd00 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient(hunting): authenticate will time out at 789.877284s 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80b77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f80b809ac40 0x7f80b81daa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56634/0 (socket says 172.21.15.110:56634) 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80b77fe640 1 -- 172.21.15.110:0/3521462535 learned_addr learned my addr 172.21.15.110:0/3521462535 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter ms_handle_connect 0x7f80b809ac40 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227463147 0 0) 0x7f80b80b1520 con 0x7f80b809ac40 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4004eb0 con 0x7f80b809ac40 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter ms_handle_connect 0x7f80b80acd00 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter ms_handle_connect 0x7f80b80b0320 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436726790 0 0) 0x7f80b80af130 con 0x7f80b80acd00 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b80b1520 con 0x7f80b80acd00 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029942824 0 0) 0x7f80b81da5a0 con 0x7f80b80b0320 2024-03-29T11:08:29.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b80af130 con 0x7f80b80b0320 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396574726 0 0) 0x7f80a4004eb0 con 0x7f80b809ac40 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80b81da5a0 con 0x7f80b809ac40 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80a8001980 con 0x7f80b809ac40 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): dump: 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 333253978 0 0) 0x7f80b81da5a0 con 0x7f80b809ac40 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 >> v1:172.21.15.105:6790/0 conn(0x7f80b80acd00 legacy=0x7f80b80afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 >> v1:172.21.15.105:6789/0 conn(0x7f80b80b0320 legacy=0x7f80b81e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 monclient: found mon.b 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b81db140 con 0x7f80b809ac40 2024-03-29T11:08:29.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient: _finish_auth 0 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f809ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _renew_subs 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b81e30f0 con 0x7f80b809ac40 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _renew_subs 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b81e3460 con 0x7f80b809ac40 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.686+0000 7f80bddc8640 1 librados: init done 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80a8001c40 con 0x7f80b809ac40 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:29.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80a8003a60 con 0x7f80b809ac40 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f80b5ffb640 4 set_mon_vals no callback set 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 10 monclient: dump: 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f80a8004010 con 0x7f80b809ac40 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1689961013 0 0) 0x7f80a8006590 con 0x7f80b809ac40 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 10 client.4120.objecter ms_dispatch 0x7f80b8000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f809ffff640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f80bddc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f80bddc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f80bddc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.687+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8098005080 con 0x7f80b809ac40 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.689+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f80a80057c0 con 0x7f80b809ac40 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.689+0000 7f809ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.689+0000 7f809ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:29.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.762+0000 7f80bddc8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2", "id": 0}] 2024-03-29T11:08:29.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.762+0000 7f80bddc8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2", "id": 0}] 2024-03-29T11:08:29.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.762+0000 7f80bddc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.762+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2", "id": 0} v 0) v1 -- 0x7f80980058b0 con 0x7f80b809ac40 2024-03-29T11:08:29.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f809ffff640 1 -- 172.21.15.110:0/3521462535 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3933209256 0 732707515) 0x7f80a8005960 con 0x7f80b809ac40 2024-03-29T11:08:29.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f809ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2", "id": 0}] 2024-03-29T11:08:29.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f809ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:29.744 INFO:teuthology.orchestra.run.smithi110.stdout:0 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 10 librados: watch_flush enter 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 10 librados: watch_flush exit 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 10 monclient: shutdown 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 >> v1:172.21.15.110:6789/0 conn(0x7f80b809ac40 legacy=0x7f80b81daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 shutdown_connections 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.765+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 >> 172.21.15.110:0/3521462535 conn(0x7f80b8066b00 msgr2=0x7f80b813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:29.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.766+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 shutdown_connections 2024-03-29T11:08:29.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.766+0000 7f80bddc8640 1 -- 172.21.15.110:0/3521462535 wait complete. 2024-03-29T11:08:29.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.766+0000 7f80bddc8640 1 librados: shutdown 2024-03-29T11:08:29.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new c5ed9abe-4bfe-4483-99c8-b39191206545 1 2024-03-29T11:08:29.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.875+0000 7f453b53d640 10 monclient: get_monmap_and_config 2024-03-29T11:08:29.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.875+0000 7f453b53d640 10 monclient: build_initial_monmap 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.875+0000 7f453b53d640 10 monclient: monmap: 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:29.877090+0000 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:29.877090+0000 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:29.855 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.876+0000 7f453b53d640 1 Processor -- start 2024-03-29T11:08:29.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.876+0000 7f453b53d640 1 -- start start 2024-03-29T11:08:29.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.876+0000 7f453b53d640 10 monclient: init 2024-03-29T11:08:29.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-a con 0x7f453409ac40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-b con 0x7f453409c870 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-c con 0x7f45340b4360 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453409dec0 con 0x7f453409ac40 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453409da30 con 0x7f45340b4360 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45340b39c0 con 0x7f453409c870 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient(hunting): authenticate will time out at 790.068251s 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f4539ada640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f45340b4360 0x7f453414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56640/0 (socket says 172.21.15.110:56640) 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f4539ada640 1 -- 172.21.15.110:0/267440112 learned_addr learned my addr 172.21.15.110:0/267440112 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349908546 0 0) 0x7f453409da30 con 0x7f45340b4360 2024-03-29T11:08:29.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4520003520 con 0x7f45340b4360 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640034267 0 0) 0x7f4524001110 con 0x7f453409ac40 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453409da30 con 0x7f453409ac40 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449292820 0 0) 0x7f45340b39c0 con 0x7f453409c870 2024-03-29T11:08:29.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4524001110 con 0x7f453409c870 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640341638 0 0) 0x7f4520003520 con 0x7f45340b4360 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 >> v1:172.21.15.105:6790/0 conn(0x7f453409c870 legacy=0x7f453413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 >> v1:172.21.15.105:6789/0 conn(0x7f453409ac40 legacy=0x7f45340b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 monclient: found mon.noname-c 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340b5160 con 0x7f45340b4360 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: _finish_auth 0 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f452c001a10 con 0x7f45340b4360 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: dump: 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f452c001bf0 con 0x7f45340b4360 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f453b53d640 4 set_mon_vals no callback set 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 1 -- 172.21.15.110:0/267440112 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f452c002000 con 0x7f45340b4360 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.877+0000 7f452bfff640 10 monclient: dump: 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 10 monclient: shutdown 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 1 -- 172.21.15.110:0/267440112 >> v1:172.21.15.110:6789/0 conn(0x7f45340b4360 legacy=0x7f453414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 1 -- 172.21.15.110:0/267440112 shutdown_connections 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 1 -- 172.21.15.110:0/267440112 >> 172.21.15.110:0/267440112 conn(0x7f4534066b00 msgr2=0x7f4534066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 1 -- 172.21.15.110:0/267440112 shutdown_connections 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.878+0000 7f453b53d640 1 -- 172.21.15.110:0/267440112 wait complete. 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 10 monclient: build_initial_monmap 2024-03-29T11:08:29.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 10 monclient: monmap: 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:29.872 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 librados: starting msgr at 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 librados: starting objecter 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 Processor -- start 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 -- start start 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 librados: setting wanted keys 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 1 librados: calling monclient init 2024-03-29T11:08:29.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.879+0000 7f453b53d640 10 monclient: init 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-c con 0x7f453409ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-a con 0x7f4534142c20 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient(hunting): picked mon.noname-b con 0x7f45341d7dc0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45341d4d30 con 0x7f4534142c20 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45341430b0 con 0x7f45341d7dc0 2024-03-29T11:08:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45341d71b0 con 0x7f453409ac40 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 10 monclient(hunting): authenticate will time out at 790.071290s 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4539ada640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f45341d7dc0 0x7f4534145a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56654/0 (socket says 172.21.15.110:56654) 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4539ada640 1 -- 172.21.15.110:0/3688232397 learned_addr learned my addr 172.21.15.110:0/3688232397 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter ms_handle_connect 0x7f45341d7dc0 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter ms_handle_connect 0x7f4534142c20 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter ms_handle_connect 0x7f453409ac40 2024-03-29T11:08:29.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905052935 0 0) 0x7f45341430b0 con 0x7f45341d7dc0 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f45341d7dc0 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528681574 0 0) 0x7f45341d4d30 con 0x7f4534142c20 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45341430b0 con 0x7f4534142c20 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671142211 0 0) 0x7f45341d71b0 con 0x7f453409ac40 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45341d4d30 con 0x7f453409ac40 2024-03-29T11:08:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4118927757 0 0) 0x7f4518004eb0 con 0x7f45341d7dc0 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45341d71b0 con 0x7f45341d7dc0 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f452c001f40 con 0x7f45341d7dc0 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): dump: 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216273598 0 0) 0x7f45341430b0 con 0x7f4534142c20 2024-03-29T11:08:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f4534142c20 2024-03-29T11:08:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1158289536 0 0) 0x7f45341d4d30 con 0x7f453409ac40 2024-03-29T11:08:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45341430b0 con 0x7f453409ac40 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4530001830 con 0x7f4534142c20 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): dump: 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f45240018a0 con 0x7f453409ac40 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient(hunting): dump: 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4114304826 0 0) 0x7f45341d71b0 con 0x7f45341d7dc0 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 >> v1:172.21.15.105:6790/0 conn(0x7f453409ac40 legacy=0x7f4534143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 >> v1:172.21.15.105:6789/0 conn(0x7f4534142c20 legacy=0x7f45341d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 monclient: found mon.b 2024-03-29T11:08:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534144ce0 con 0x7f45341d7dc0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient: _finish_auth 0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f4529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.880+0000 7f453b53d640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 10 monclient: _renew_subs 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4534145490 con 0x7f45341d7dc0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f452c004e10 con 0x7f45341d7dc0 2024-03-29T11:08:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f452c005120 con 0x7f45341d7dc0 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 10 monclient: dump: 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:29.886 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 10 monclient: _renew_subs 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f452b7fe640 4 set_mon_vals no callback set 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4534146330 con 0x7f45341d7dc0 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f453b53d640 1 librados: init done 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f452c005610 con 0x7f45341d7dc0 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2455142456 0 0) 0x7f452c005e50 con 0x7f45341d7dc0 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 10 client.4126.objecter ms_dispatch 0x7f4534000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:08:29.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.881+0000 7f4529ffb640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.882+0000 7f453b53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.882+0000 7f453b53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.882+0000 7f453b53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.882+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4534145680 con 0x7f45341d7dc0 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.884+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f452c006090 con 0x7f45341d7dc0 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.884+0000 7f4529ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.884+0000 7f4529ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.956+0000 7f453b53d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c5ed9abe-4bfe-4483-99c8-b39191206545", "id": 1}] 2024-03-29T11:08:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.956+0000 7f453b53d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c5ed9abe-4bfe-4483-99c8-b39191206545", "id": 1}] 2024-03-29T11:08:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.956+0000 7f453b53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.956+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c5ed9abe-4bfe-4483-99c8-b39191206545", "id": 1} v 0) v1 -- 0x7f453409b020 con 0x7f45341d7dc0 2024-03-29T11:08:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.959+0000 7f4529ffb640 1 -- 172.21.15.110:0/3688232397 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5ed9abe-4bfe-4483-99c8-b39191206545", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3009672151 0 940483276) 0x7f452c006850 con 0x7f45341d7dc0 2024-03-29T11:08:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.959+0000 7f4529ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c5ed9abe-4bfe-4483-99c8-b39191206545", "id": 1}] 2024-03-29T11:08:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.959+0000 7f4529ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:29.939 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-03-29T11:08:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 10 librados: watch_flush enter 2024-03-29T11:08:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 10 librados: watch_flush exit 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 10 monclient: shutdown 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 >> v1:172.21.15.110:6789/0 conn(0x7f45341d7dc0 legacy=0x7f4534145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 shutdown_connections 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 >> 172.21.15.110:0/3688232397 conn(0x7f4534066b00 msgr2=0x7f453413a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 shutdown_connections 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 -- 172.21.15.110:0/3688232397 wait complete. 2024-03-29T11:08:29.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:29.960+0000 7f453b53d640 1 librados: shutdown 2024-03-29T11:08:29.950 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 34c35504-e154-448d-83c3-102515f1ea0b 2 2024-03-29T11:08:30.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.075+0000 7f2b2d8e6640 10 monclient: get_monmap_and_config 2024-03-29T11:08:30.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.075+0000 7f2b2d8e6640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.075+0000 7f2b2d8e6640 10 monclient: monmap: 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:30.076985+0000 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:30.076985+0000 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:30.055 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 1 Processor -- start 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 1 -- start start 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient: init 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b280b8240 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b280b1f60 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2809c8c0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2809bd80 con 0x7f2b280b8240 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2809b960 con 0x7f2b280b1f60 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280b2ec0 con 0x7f2b2809c8c0 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.076+0000 7f2b2d8e6640 10 monclient(hunting): authenticate will time out at 790.268052s 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b267fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2b280b1f60 0x7f2b2809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56662/0 (socket says 172.21.15.110:56662) 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b267fc640 1 -- 172.21.15.110:0/3168836295 learned_addr learned my addr 172.21.15.110:0/3168836295 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313147528 0 0) 0x7f2b2809b960 con 0x7f2b280b1f60 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b10003520 con 0x7f2b280b1f60 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766090363 0 0) 0x7f2b280b2ec0 con 0x7f2b2809c8c0 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b2809b960 con 0x7f2b2809c8c0 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126238958 0 0) 0x7f2b2809bd80 con 0x7f2b280b8240 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b280b2ec0 con 0x7f2b280b8240 2024-03-29T11:08:30.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3235028865 0 0) 0x7f2b10003520 con 0x7f2b280b1f60 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 >> v1:172.21.15.105:6790/0 conn(0x7f2b2809c8c0 legacy=0x7f2b2814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 >> v1:172.21.15.105:6789/0 conn(0x7f2b280b8240 legacy=0x7f2b2813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 monclient: found mon.noname-c 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2813fb60 con 0x7f2b280b1f60 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b1c001690 con 0x7f2b280b1f60 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b2d8e6640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: dump: 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b2d8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1c0017f0 con 0x7f2b280b1f60 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 1 -- 172.21.15.110:0/3168836295 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b1c001c00 con 0x7f2b280b1f60 2024-03-29T11:08:30.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b2d8e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b2d8e6640 4 set_mon_vals no callback set 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.077+0000 7f2b25ffb640 10 monclient: dump: 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 10 monclient: shutdown 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/3168836295 >> v1:172.21.15.110:6789/0 conn(0x7f2b280b1f60 legacy=0x7f2b2809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/3168836295 shutdown_connections 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/3168836295 >> 172.21.15.110:0/3168836295 conn(0x7f2b28066b00 msgr2=0x7f2b28066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/3168836295 shutdown_connections 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/3168836295 wait complete. 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.078+0000 7f2b2d8e6640 10 monclient: monmap: 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.071 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 librados: starting msgr at 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 librados: starting objecter 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 Processor -- start 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 -- start start 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 librados: setting wanted keys 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 librados: calling monclient init 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient: init 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2809c8c0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f2b280b8240 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b280ae170 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280af170 con 0x7f2b280ae170 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.079+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280ae620 con 0x7f2b280b8240 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280b0360 con 0x7f2b2809c8c0 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient(hunting): authenticate will time out at 790.271133s 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b267fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2b280b8240 0x7f2b281d7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56676/0 (socket says 172.21.15.110:56676) 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b267fc640 1 -- 172.21.15.110:0/2433439199 learned_addr learned my addr 172.21.15.110:0/2433439199 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter ms_handle_connect 0x7f2b280b8240 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter ms_handle_connect 0x7f2b2809c8c0 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter ms_handle_connect 0x7f2b280ae170 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366615438 0 0) 0x7f2b280ae620 con 0x7f2b280b8240 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b14004eb0 con 0x7f2b280b8240 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155754387 0 0) 0x7f2b280b0360 con 0x7f2b2809c8c0 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b280ae620 con 0x7f2b2809c8c0 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490614553 0 0) 0x7f2b280af170 con 0x7f2b280ae170 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b280b0360 con 0x7f2b280ae170 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1380266901 0 0) 0x7f2b14004eb0 con 0x7f2b280b8240 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b280af170 con 0x7f2b280b8240 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b1c001b40 con 0x7f2b280b8240 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448866055 0 0) 0x7f2b280ae620 con 0x7f2b2809c8c0 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b14004eb0 con 0x7f2b2809c8c0 2024-03-29T11:08:30.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b180017d0 con 0x7f2b2809c8c0 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358857812 0 0) 0x7f2b280b0360 con 0x7f2b280ae170 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b280ae620 con 0x7f2b280ae170 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b20001b60 con 0x7f2b280ae170 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.078 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 651493132 0 0) 0x7f2b280af170 con 0x7f2b280b8240 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 >> v1:172.21.15.105:6790/0 conn(0x7f2b2809c8c0 legacy=0x7f2b280aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 >> v1:172.21.15.105:6789/0 conn(0x7f2b280ae170 legacy=0x7f2b281e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 monclient: found mon.b 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b281d4e60 con 0x7f2b280b8240 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient: _renew_subs 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b281d7330 con 0x7f2b280b8240 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient: _renew_subs 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b281e3150 con 0x7f2b280b8240 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b2d8e6640 1 librados: init done 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1c004a90 con 0x7f2b280b8240 2024-03-29T11:08:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2b1c004da0 con 0x7f2b280b8240 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b257fa640 4 set_mon_vals no callback set 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.080+0000 7f2b0ffff640 10 monclient: dump: 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2b1c005290 con 0x7f2b280b8240 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 4032956448 0 0) 0x7f2b1c006890 con 0x7f2b280b8240 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 10 client.4132.objecter ms_dispatch 0x7f2b28000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b0ffff640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b2d8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b2d8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b2d8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.081+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08005080 con 0x7f2b280b8240 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.083+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2b1c006470 con 0x7f2b280b8240 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.083+0000 7f2b0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.083+0000 7f2b0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:30.083 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.085+0000 7f8921e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:08:30.091 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.112+0000 7fd612032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:08:30.121 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.141+0000 7f8921e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:08:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.156+0000 7f2b2d8e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "34c35504-e154-448d-83c3-102515f1ea0b", "id": 2}] 2024-03-29T11:08:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.156+0000 7f2b2d8e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "34c35504-e154-448d-83c3-102515f1ea0b", "id": 2}] 2024-03-29T11:08:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.156+0000 7f2b2d8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.156+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "34c35504-e154-448d-83c3-102515f1ea0b", "id": 2} v 0) v1 -- 0x7f2b080058b0 con 0x7f2b280b8240 2024-03-29T11:08:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b0ffff640 1 -- 172.21.15.110:0/2433439199 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34c35504-e154-448d-83c3-102515f1ea0b", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4033748612 0 216599125) 0x7f2b1c006aa0 con 0x7f2b280b8240 2024-03-29T11:08:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "34c35504-e154-448d-83c3-102515f1ea0b", "id": 2}] 2024-03-29T11:08:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:30.138 INFO:teuthology.orchestra.run.smithi110.stdout:2 2024-03-29T11:08:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 10 librados: watch_flush enter 2024-03-29T11:08:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 10 librados: watch_flush exit 2024-03-29T11:08:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 10 monclient: shutdown 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 >> v1:172.21.15.110:6789/0 conn(0x7f2b280b8240 legacy=0x7f2b281d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 shutdown_connections 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.159+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 >> 172.21.15.110:0/2433439199 conn(0x7f2b28066b00 msgr2=0x7f2b2806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.160+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 shutdown_connections 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.160+0000 7f2b2d8e6640 1 -- 172.21.15.110:0/2433439199 wait complete. 2024-03-29T11:08:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.160+0000 7f2b2d8e6640 1 librados: shutdown 2024-03-29T11:08:30.145 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.167+0000 7fd612032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:08:30.149 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new e8cf0178-6e7e-43bb-a995-fcd094aa4688 3 2024-03-29T11:08:30.167 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.188+0000 7f8921e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:08:30.190 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.211+0000 7fd612032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:08:30.224 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.245+0000 7f8921e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:08:30.243 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.264+0000 7fd612032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.274+0000 7fed99f9f640 10 monclient: get_monmap_and_config 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.274+0000 7fed99f9f640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.274+0000 7fed99f9f640 10 monclient: monmap: 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:30.275989+0000 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:30.275989+0000 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:30.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 1 Processor -- start 2024-03-29T11:08:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 1 -- start start 2024-03-29T11:08:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient: init 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-a con 0x7fed94145610 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-c con 0x7fed94146050 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.275+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-b con 0x7fed9413c030 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940aacc0 con 0x7fed94145610 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940b2720 con 0x7fed94146050 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940aaf30 con 0x7fed9413c030 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 10 monclient(hunting): authenticate will time out at 790.467142s 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed93fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fed94146050 0x7fed9413b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56682/0 (socket says 172.21.15.110:56682) 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed93fff640 1 -- 172.21.15.110:0/1310081113 learned_addr learned my addr 172.21.15.110:0/1310081113 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152781906 0 0) 0x7fed940b2720 con 0x7fed94146050 2024-03-29T11:08:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84003520 con 0x7fed94146050 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771773769 0 0) 0x7fed940aaf30 con 0x7fed9413c030 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed940b2720 con 0x7fed9413c030 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487890626 0 0) 0x7fed940aacc0 con 0x7fed94145610 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed940aaf30 con 0x7fed94145610 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677465223 0 0) 0x7fed84003520 con 0x7fed94146050 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 >> v1:172.21.15.105:6790/0 conn(0x7fed9413c030 legacy=0x7fed9414d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 >> v1:172.21.15.105:6789/0 conn(0x7fed94145610 legacy=0x7fed940b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 monclient: found mon.noname-c 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed940a9b20 con 0x7fed94146050 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed80001690 con 0x7fed94146050 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: dump: 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed80001870 con 0x7fed94146050 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 1 -- 172.21.15.110:0/1310081113 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed80001c00 con 0x7fed94146050 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed99f9f640 4 set_mon_vals no callback set 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.276+0000 7fed937fe640 10 monclient: dump: 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 10 monclient: shutdown 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 1 -- 172.21.15.110:0/1310081113 >> v1:172.21.15.110:6789/0 conn(0x7fed94146050 legacy=0x7fed9413b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 1 -- 172.21.15.110:0/1310081113 shutdown_connections 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 1 -- 172.21.15.110:0/1310081113 >> 172.21.15.110:0/1310081113 conn(0x7fed940726e0 msgr2=0x7fed94066920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 1 -- 172.21.15.110:0/1310081113 shutdown_connections 2024-03-29T11:08:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.277+0000 7fed99f9f640 1 -- 172.21.15.110:0/1310081113 wait complete. 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 10 monclient: monmap: 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 librados: starting msgr at 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 librados: starting objecter 2024-03-29T11:08:30.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 Processor -- start 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 -- start start 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 librados: setting wanted keys 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 1 librados: calling monclient init 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.278+0000 7fed99f9f640 10 monclient: init 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-a con 0x7fed9413c030 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-b con 0x7fed94145610 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient(hunting): picked mon.noname-c con 0x7fed94146050 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9413e370 con 0x7fed9413c030 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed941d7fa0 con 0x7fed94145610 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed941d7090 con 0x7fed94146050 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed99f9f640 10 monclient(hunting): authenticate will time out at 790.470302s 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed93fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fed94145610 0x7fed941d7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56698/0 (socket says 172.21.15.110:56698) 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed93fff640 1 -- 172.21.15.110:0/4201252837 learned_addr learned my addr 172.21.15.110:0/4201252837 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter ms_handle_connect 0x7fed94145610 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter ms_handle_connect 0x7fed9413c030 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter ms_handle_connect 0x7fed94146050 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194930456 0 0) 0x7fed941d7fa0 con 0x7fed94145610 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:08:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed78004eb0 con 0x7fed94145610 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777197289 0 0) 0x7fed9413e370 con 0x7fed9413c030 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed941d7fa0 con 0x7fed9413c030 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280042360 0 0) 0x7fed941d7090 con 0x7fed94146050 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:08:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9413e370 con 0x7fed94146050 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237354685 0 0) 0x7fed78004eb0 con 0x7fed94145610 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed941d7090 con 0x7fed94145610 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254598625 0 0) 0x7fed941d7fa0 con 0x7fed9413c030 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed78004eb0 con 0x7fed9413c030 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed80001c40 con 0x7fed94145610 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): dump: 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2979660607 0 0) 0x7fed9413e370 con 0x7fed94146050 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed941d7fa0 con 0x7fed94146050 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed88001830 con 0x7fed9413c030 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): dump: 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed8c001c30 con 0x7fed94146050 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.279+0000 7fed917fa640 10 monclient(hunting): dump: 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1552850713 0 0) 0x7fed941d7090 con 0x7fed94145610 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 >> v1:172.21.15.105:6790/0 conn(0x7fed94146050 legacy=0x7fed941e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 >> v1:172.21.15.105:6789/0 conn(0x7fed9413c030 legacy=0x7fed9413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 monclient: found mon.b 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed941d7370 con 0x7fed94145610 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 10 monclient: _renew_subs 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed941d4c10 con 0x7fed94145610 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 10 monclient: _renew_subs 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9413e5e0 con 0x7fed94145610 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed99f9f640 1 librados: init done 2024-03-29T11:08:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed800011f0 con 0x7fed94145610 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed92ffd640 4 set_mon_vals no callback set 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fed80004c30 con 0x7fed94145610 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 monclient: dump: 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed80005220 con 0x7fed94145610 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2302718719 0 0) 0x7fed800064a0 con 0x7fed94145610 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 10 client.4138.objecter ms_dispatch 0x7fed94000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.280+0000 7fed917fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.281+0000 7fed99f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.281+0000 7fed99f9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.281+0000 7fed99f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.281+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed74005080 con 0x7fed94145610 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.283+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed800066d0 con 0x7fed94145610 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.283+0000 7fed917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.283+0000 7fed917fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:30.282 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.291+0000 7f8921e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:08:30.286 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.308+0000 7fd612032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:08:30.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.355+0000 7fed99f9f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e8cf0178-6e7e-43bb-a995-fcd094aa4688", "id": 3}] 2024-03-29T11:08:30.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.355+0000 7fed99f9f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e8cf0178-6e7e-43bb-a995-fcd094aa4688", "id": 3}] 2024-03-29T11:08:30.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.355+0000 7fed99f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.355+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e8cf0178-6e7e-43bb-a995-fcd094aa4688", "id": 3} v 0) v1 -- 0x7fed740058b0 con 0x7fed94145610 2024-03-29T11:08:30.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.358+0000 7fed917fa640 1 -- 172.21.15.110:0/4201252837 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8cf0178-6e7e-43bb-a995-fcd094aa4688", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2638361622 0 525046306) 0x7fed80006930 con 0x7fed94145610 2024-03-29T11:08:30.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.358+0000 7fed917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e8cf0178-6e7e-43bb-a995-fcd094aa4688", "id": 3}] 2024-03-29T11:08:30.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.358+0000 7fed917fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:30.337 INFO:teuthology.orchestra.run.smithi110.stdout:3 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 10 librados: watch_flush enter 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 10 librados: watch_flush exit 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 10 monclient: shutdown 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 >> v1:172.21.15.110:6789/0 conn(0x7fed94145610 legacy=0x7fed941d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 shutdown_connections 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 >> 172.21.15.110:0/4201252837 conn(0x7fed940726e0 msgr2=0x7fed940b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 shutdown_connections 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 -- 172.21.15.110:0/4201252837 wait complete. 2024-03-29T11:08:30.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.359+0000 7fed99f9f640 1 librados: shutdown 2024-03-29T11:08:30.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 7b2ae2cc-80b4-4217-a8b8-700093947982 4 2024-03-29T11:08:30.390 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.411+0000 7f8921e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:08:30.400 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.422+0000 7fd612032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:08:30.464 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.485+0000 7f8921e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:08:30.470 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.491+0000 7fd612032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:08:30.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.510+0000 7f549cd82640 10 monclient: get_monmap_and_config 2024-03-29T11:08:30.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.510+0000 7f549cd82640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.510+0000 7f549cd82640 10 monclient: monmap: 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:30.512231+0000 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:30.512231+0000 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:30.490 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.511+0000 7f549cd82640 1 Processor -- start 2024-03-29T11:08:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.511+0000 7f549cd82640 1 -- start start 2024-03-29T11:08:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.511+0000 7f549cd82640 10 monclient: init 2024-03-29T11:08:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-b con 0x7f549809ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-a con 0x7f549809c870 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-c con 0x7f54980b4360 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549809dec0 con 0x7f549809c870 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549809da30 con 0x7f54980b4360 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980b39c0 con 0x7f549809ac40 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549cd82640 10 monclient(hunting): authenticate will time out at 790.703501s 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f5496d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f54980b4360 0x7f549814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56702/0 (socket says 172.21.15.110:56702) 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f5496d9d640 1 -- 172.21.15.110:0/4187110561 learned_addr learned my addr 172.21.15.110:0/4187110561 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082822044 0 0) 0x7f549809da30 con 0x7f54980b4360 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484003520 con 0x7f54980b4360 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533962805 0 0) 0x7f54980b39c0 con 0x7f549809ac40 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549809da30 con 0x7f549809ac40 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181626779 0 0) 0x7f5488001110 con 0x7f549809c870 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54980b39c0 con 0x7f549809c870 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1662214224 0 0) 0x7f5484003520 con 0x7f54980b4360 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 >> v1:172.21.15.105:6790/0 conn(0x7f549809ac40 legacy=0x7f54980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 >> v1:172.21.15.105:6789/0 conn(0x7f549809c870 legacy=0x7f549813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 monclient: found mon.noname-c 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980b5160 con 0x7f54980b4360 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.512+0000 7f549559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5490001a10 con 0x7f54980b4360 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: dump: 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5490001bf0 con 0x7f54980b4360 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 1 -- 172.21.15.110:0/4187110561 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5490002000 con 0x7f54980b4360 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 4 set_mon_vals no callback set 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549559a640 10 monclient: dump: 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 10 monclient: shutdown 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 1 -- 172.21.15.110:0/4187110561 >> v1:172.21.15.110:6789/0 conn(0x7f54980b4360 legacy=0x7f549814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 1 -- 172.21.15.110:0/4187110561 shutdown_connections 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 1 -- 172.21.15.110:0/4187110561 >> 172.21.15.110:0/4187110561 conn(0x7f5498066b00 msgr2=0x7f5498066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 1 -- 172.21.15.110:0/4187110561 shutdown_connections 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.513+0000 7f549cd82640 1 -- 172.21.15.110:0/4187110561 wait complete. 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 10 monclient: monmap: 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 librados: starting msgr at 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 librados: starting objecter 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 Processor -- start 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 -- start start 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 librados: setting wanted keys 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 1 librados: calling monclient init 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.514+0000 7f549cd82640 10 monclient: init 2024-03-29T11:08:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-c con 0x7f549809ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-b con 0x7f54980b7310 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient(hunting): picked mon.noname-a con 0x7f54981d7de0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980b78c0 con 0x7f54981d7de0 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980b7d50 con 0x7f54980b7310 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5498146480 con 0x7f549809ac40 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f549cd82640 10 monclient(hunting): authenticate will time out at 790.706537s 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f5495d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f54980b7310 0x7f54981d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56716/0 (socket says 172.21.15.110:56716) 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f5495d9b640 1 -- 172.21.15.110:0/3863349413 learned_addr learned my addr 172.21.15.110:0/3863349413 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter ms_handle_connect 0x7f54980b7310 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter ms_handle_connect 0x7f549809ac40 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter ms_handle_connect 0x7f54981d7de0 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500845527 0 0) 0x7f54980b7d50 con 0x7f54980b7310 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478004eb0 con 0x7f54980b7310 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520492044 0 0) 0x7f5498146480 con 0x7f549809ac40 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54980b7d50 con 0x7f549809ac40 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287063398 0 0) 0x7f54980b78c0 con 0x7f54981d7de0 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.515+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5498146480 con 0x7f54981d7de0 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3024648897 0 0) 0x7f54980b7d50 con 0x7f549809ac40 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54980b78c0 con 0x7f549809ac40 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5480001930 con 0x7f549809ac40 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): dump: 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165910845 0 0) 0x7f5478004eb0 con 0x7f54980b7310 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54980b7d50 con 0x7f54980b7310 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553342677 0 0) 0x7f5498146480 con 0x7f54981d7de0 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5478004eb0 con 0x7f54981d7de0 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5488001860 con 0x7f54980b7310 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): dump: 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5490001d60 con 0x7f54981d7de0 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient(hunting): dump: 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2416391501 0 0) 0x7f54980b78c0 con 0x7f549809ac40 2024-03-29T11:08:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 >> v1:172.21.15.110:6789/0 conn(0x7f54980b7310 legacy=0x7f54981d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 >> v1:172.21.15.105:6789/0 conn(0x7f54981d7de0 legacy=0x7f54981e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 monclient: found mon.c 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54981e68d0 con 0x7f549809ac40 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 5 monclient: authenticate success, global_id 4145 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 10 monclient: _renew_subs 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54981e38a0 con 0x7f549809ac40 2024-03-29T11:08:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 10 monclient: _renew_subs 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54981e3d80 con 0x7f549809ac40 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f549cd82640 1 librados: init done 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5480001d40 con 0x7f549809ac40 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5480003790 con 0x7f549809ac40 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f5494d99640 4 set_mon_vals no callback set 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 monclient: dump: 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5480003d40 con 0x7f549809ac40 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3423974270 0 0) 0x7f54800054c0 con 0x7f549809ac40 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 10 client.4145.objecter ms_dispatch 0x7f5498000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 3 client.4145.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 3 client.4145.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.516+0000 7f548effd640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.517+0000 7f549cd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.517+0000 7f549cd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.517+0000 7f549cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:08:30.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.517+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c005080 con 0x7f549809ac40 2024-03-29T11:08:30.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.519+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5480004fb0 con 0x7f549809ac40 2024-03-29T11:08:30.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.519+0000 7f548effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.519+0000 7f548effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:30.519 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.534+0000 7f8921e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:08:30.519 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.539+0000 7fd612032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:08:30.567 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.588+0000 7f8921e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:08:30.569 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.591+0000 7fd612032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:08:30.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.593+0000 7f549cd82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7b2ae2cc-80b4-4217-a8b8-700093947982", "id": 4}] 2024-03-29T11:08:30.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.593+0000 7f549cd82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7b2ae2cc-80b4-4217-a8b8-700093947982", "id": 4}] 2024-03-29T11:08:30.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.593+0000 7f549cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:08:30.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.593+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd new", "uuid": "7b2ae2cc-80b4-4217-a8b8-700093947982", "id": 4} v 0) v1 -- 0x7f547c0058b0 con 0x7f549809ac40 2024-03-29T11:08:30.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f548effd640 1 -- 172.21.15.110:0/3863349413 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b2ae2cc-80b4-4217-a8b8-700093947982", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1778719320 0 1697012583) 0x7f5480005150 con 0x7f549809ac40 2024-03-29T11:08:30.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f548effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7b2ae2cc-80b4-4217-a8b8-700093947982", "id": 4}] 2024-03-29T11:08:30.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f548effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:30.575 INFO:teuthology.orchestra.run.smithi110.stdout:4 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f549cd82640 10 librados: watch_flush enter 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f549cd82640 10 librados: watch_flush exit 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f549cd82640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.596+0000 7f549cd82640 10 client.4145.objecter successfully canceled tick 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 10 monclient: shutdown 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 >> v1:172.21.15.105:6790/0 conn(0x7f549809ac40 legacy=0x7f54980b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 shutdown_connections 2024-03-29T11:08:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 >> 172.21.15.110:0/3863349413 conn(0x7f5498066b00 msgr2=0x7f549813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 shutdown_connections 2024-03-29T11:08:30.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 -- 172.21.15.110:0/3863349413 wait complete. 2024-03-29T11:08:30.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.597+0000 7f549cd82640 1 librados: shutdown 2024-03-29T11:08:30.586 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new d31c42e4-e9c2-467f-85a3-ee252451edf4 5 2024-03-29T11:08:30.616 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:30.638+0000 7fd612032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:08:30.618 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:30.639+0000 7f8921e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:08:30.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.705+0000 7f14040f6640 10 monclient: get_monmap_and_config 2024-03-29T11:08:30.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.705+0000 7f14040f6640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.705+0000 7f14040f6640 10 monclient: monmap: 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:30.706823+0000 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:30.706823+0000 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 1 Processor -- start 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 1 -- start start 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient: init 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-c con 0x7f13fc09ac40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-a con 0x7f13fc09c870 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-b con 0x7f13fc0b4360 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc09dec0 con 0x7f13fc09c870 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc09da30 con 0x7f13fc09ac40 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc0b39c0 con 0x7f13fc0b4360 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f14040f6640 10 monclient(hunting): authenticate will time out at 790.897977s 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f1401e92640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f13fc09ac40 0x7f13fc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56718/0 (socket says 172.21.15.110:56718) 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.706+0000 7f1401e92640 1 -- 172.21.15.110:0/360447731 learned_addr learned my addr 172.21.15.110:0/360447731 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342757224 0 0) 0x7f13fc09da30 con 0x7f13fc09ac40 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e8003520 con 0x7f13fc09ac40 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148044110 0 0) 0x7f13fc0b39c0 con 0x7f13fc0b4360 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc09da30 con 0x7f13fc0b4360 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136356731 0 0) 0x7f13ec001110 con 0x7f13fc09c870 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc0b39c0 con 0x7f13fc09c870 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3245314329 0 0) 0x7f13e8003520 con 0x7f13fc09ac40 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 >> v1:172.21.15.105:6790/0 conn(0x7f13fc0b4360 legacy=0x7f13fc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 >> v1:172.21.15.105:6789/0 conn(0x7f13fc09c870 legacy=0x7f13fc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 monclient: found mon.noname-c 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc0b5160 con 0x7f13fc09ac40 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13f8001690 con 0x7f13fc09ac40 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f14040f6640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f14040f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: dump: 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f14040f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13f8001870 con 0x7f13fc09ac40 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 1 -- 172.21.15.110:0/360447731 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13f8001c80 con 0x7f13fc09ac40 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f14040f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f14040f6640 4 set_mon_vals no callback set 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.707+0000 7f1400e90640 10 monclient: dump: 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 10 monclient: shutdown 2024-03-29T11:08:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 1 -- 172.21.15.110:0/360447731 >> v1:172.21.15.110:6789/0 conn(0x7f13fc09ac40 legacy=0x7f13fc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 1 -- 172.21.15.110:0/360447731 shutdown_connections 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 1 -- 172.21.15.110:0/360447731 >> 172.21.15.110:0/360447731 conn(0x7f13fc066b00 msgr2=0x7f13fc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 1 -- 172.21.15.110:0/360447731 shutdown_connections 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 1 -- 172.21.15.110:0/360447731 wait complete. 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.708+0000 7f14040f6640 10 monclient: monmap: 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.700 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 librados: starting msgr at 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 librados: starting objecter 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 Processor -- start 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 -- start start 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 librados: setting wanted keys 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 librados: calling monclient init 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient: init 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-c con 0x7f13fc09ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-a con 0x7f13fc0ad140 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient(hunting): picked mon.noname-b con 0x7f13fc1d77e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc1d7c20 con 0x7f13fc0ad140 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc1d7db0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc1d7f40 con 0x7f13fc09ac40 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f14040f6640 10 monclient(hunting): authenticate will time out at 790.900867s 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f1402693640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f13fc1d77e0 0x7f13fc1e1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56724/0 (socket says 172.21.15.110:56724) 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f1402693640 1 -- 172.21.15.110:0/726384532 learned_addr learned my addr 172.21.15.110:0/726384532 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f13f27fc640 10 client.?.objecter ms_handle_connect 0x7f13fc1d77e0 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f13f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f13f27fc640 10 client.?.objecter ms_handle_connect 0x7f13fc0ad140 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f13f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.709+0000 7f13f27fc640 10 client.?.objecter ms_handle_connect 0x7f13fc09ac40 2024-03-29T11:08:30.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454195189 0 0) 0x7f13fc1d7db0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e0004eb0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559109048 0 0) 0x7f13fc1d7c20 con 0x7f13fc0ad140 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc1d7db0 con 0x7f13fc0ad140 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866542251 0 0) 0x7f13fc1d7f40 con 0x7f13fc09ac40 2024-03-29T11:08:30.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc1d7c20 con 0x7f13fc09ac40 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1120085664 0 0) 0x7f13e0004eb0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13fc1d7f40 con 0x7f13fc1d77e0 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13f4001c00 con 0x7f13fc1d77e0 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): dump: 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.704 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 370235099 0 0) 0x7f13fc1d7c20 con 0x7f13fc09ac40 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e0004eb0 con 0x7f13fc09ac40 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571303288 0 0) 0x7f13fc1d7db0 con 0x7f13fc0ad140 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13fc1d7c20 con 0x7f13fc0ad140 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13f8001b80 con 0x7f13fc09ac40 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): dump: 2024-03-29T11:08:30.705 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13ec001860 con 0x7f13fc0ad140 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient(hunting): dump: 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2151371661 0 0) 0x7f13fc1d7f40 con 0x7f13fc1d77e0 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 >> v1:172.21.15.105:6790/0 conn(0x7f13fc09ac40 legacy=0x7f13fc0b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 >> v1:172.21.15.105:6789/0 conn(0x7f13fc0ad140 legacy=0x7f13fc0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 monclient: found mon.b 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc1e6390 con 0x7f13fc1d77e0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 10 monclient: _renew_subs 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc1e3360 con 0x7f13fc1d77e0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 10 monclient: _renew_subs 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc1e38e0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f14040f6640 1 librados: init done 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13f40020d0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f13f4003b30 con 0x7f13fc1d77e0 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f3fff640 4 set_mon_vals no callback set 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 monclient: dump: 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.709 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f13f40040e0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2014153338 0 0) 0x7f13f4006a60 con 0x7f13fc1d77e0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 10 client.4150.objecter ms_dispatch 0x7f13fc000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.710+0000 7f13f27fc640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.711+0000 7f14040f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.711+0000 7f14040f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.711+0000 7f14040f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.711+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e4005080 con 0x7f13fc1d77e0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.713+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f13f40045d0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.713+0000 7f13f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.713+0000 7f13f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:30.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.785+0000 7f14040f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d31c42e4-e9c2-467f-85a3-ee252451edf4", "id": 5}] 2024-03-29T11:08:30.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.785+0000 7f14040f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d31c42e4-e9c2-467f-85a3-ee252451edf4", "id": 5}] 2024-03-29T11:08:30.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.785+0000 7f14040f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.785+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d31c42e4-e9c2-467f-85a3-ee252451edf4", "id": 5} v 0) v1 -- 0x7f13e40058b0 con 0x7f13fc1d77e0 2024-03-29T11:08:30.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.788+0000 7f13f27fc640 1 -- 172.21.15.110:0/726384532 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d31c42e4-e9c2-467f-85a3-ee252451edf4", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 4155226313 0 1988412176) 0x7f13f4006490 con 0x7f13fc1d77e0 2024-03-29T11:08:30.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.788+0000 7f13f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d31c42e4-e9c2-467f-85a3-ee252451edf4", "id": 5}] 2024-03-29T11:08:30.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.788+0000 7f13f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:30.767 INFO:teuthology.orchestra.run.smithi110.stdout:5 2024-03-29T11:08:30.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 10 librados: watch_flush enter 2024-03-29T11:08:30.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 10 librados: watch_flush exit 2024-03-29T11:08:30.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 10 monclient: shutdown 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 >> v1:172.21.15.110:6789/0 conn(0x7f13fc1d77e0 legacy=0x7f13fc1e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 shutdown_connections 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 >> 172.21.15.110:0/726384532 conn(0x7f13fc066b00 msgr2=0x7f13fc13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 shutdown_connections 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 -- 172.21.15.110:0/726384532 wait complete. 2024-03-29T11:08:30.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.789+0000 7f14040f6640 1 librados: shutdown 2024-03-29T11:08:30.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new f231f5e6-d6ae-48a0-8a56-52f22a494330 6 2024-03-29T11:08:30.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.941+0000 7f4d49b32640 10 monclient: get_monmap_and_config 2024-03-29T11:08:30.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.941+0000 7f4d49b32640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.941+0000 7f4d49b32640 10 monclient: monmap: 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:30.942977+0000 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:30.942977+0000 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:30.921 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 1 Processor -- start 2024-03-29T11:08:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 1 -- start start 2024-03-29T11:08:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 10 monclient: init 2024-03-29T11:08:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4409ac40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.942+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4409c870 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4d440b4360 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4409dec0 con 0x7f4d4409c870 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4409da30 con 0x7f4d4409ac40 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440b39c0 con 0x7f4d440b4360 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 10 monclient(hunting): authenticate will time out at 791.134164s 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d437fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4d4409ac40 0x7f4d440b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56740/0 (socket says 172.21.15.110:56740) 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d437fe640 1 -- 172.21.15.110:0/770894174 learned_addr learned my addr 172.21.15.110:0/770894174 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768464701 0 0) 0x7f4d4409da30 con 0x7f4d4409ac40 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c003520 con 0x7f4d4409ac40 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087777785 0 0) 0x7f4d440b39c0 con 0x7f4d440b4360 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d4409da30 con 0x7f4d440b4360 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927457948 0 0) 0x7f4d34001110 con 0x7f4d4409c870 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:08:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d440b39c0 con 0x7f4d4409c870 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269559849 0 0) 0x7f4d2c003520 con 0x7f4d4409ac40 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 >> v1:172.21.15.105:6790/0 conn(0x7f4d440b4360 legacy=0x7f4d4414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 >> v1:172.21.15.105:6789/0 conn(0x7f4d4409c870 legacy=0x7f4d4413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 monclient: found mon.noname-c 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440b5160 con 0x7f4d4409ac40 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d38001690 con 0x7f4d4409ac40 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:08:30.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: dump: 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d38001870 con 0x7f4d4409ac40 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 1 -- 172.21.15.110:0/770894174 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d38001c80 con 0x7f4d4409ac40 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d49b32640 4 set_mon_vals no callback set 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.943+0000 7f4d427fc640 10 monclient: dump: 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.934 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 10 monclient: shutdown 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 1 -- 172.21.15.110:0/770894174 >> v1:172.21.15.110:6789/0 conn(0x7f4d4409ac40 legacy=0x7f4d440b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 1 -- 172.21.15.110:0/770894174 shutdown_connections 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 1 -- 172.21.15.110:0/770894174 >> 172.21.15.110:0/770894174 conn(0x7f4d44066b00 msgr2=0x7f4d44066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 1 -- 172.21.15.110:0/770894174 shutdown_connections 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 1 -- 172.21.15.110:0/770894174 wait complete. 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 10 monclient: build_initial_monmap 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.944+0000 7f4d49b32640 10 monclient: monmap: 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 librados: starting msgr at 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 librados: starting objecter 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 Processor -- start 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 -- start start 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 librados: setting wanted keys 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 librados: calling monclient init 2024-03-29T11:08:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient: init 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4409ac40 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4d440adbe0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4d44136a40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440ae720 con 0x7f4d44136a40 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.945+0000 7f4d49b32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440af7a0 con 0x7f4d440adbe0 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440ae200 con 0x7f4d4409ac40 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient(hunting): authenticate will time out at 791.137125s 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d42ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4d440adbe0 0x7f4d44136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56746/0 (socket says 172.21.15.110:56746) 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d42ffd640 1 -- 172.21.15.110:0/3206498037 learned_addr learned my addr 172.21.15.110:0/3206498037 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter ms_handle_connect 0x7f4d440adbe0 2024-03-29T11:08:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter ms_handle_connect 0x7f4d4409ac40 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter ms_handle_connect 0x7f4d44136a40 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264373624 0 0) 0x7f4d440af7a0 con 0x7f4d440adbe0 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30004eb0 con 0x7f4d440adbe0 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936027821 0 0) 0x7f4d440ae200 con 0x7f4d4409ac40 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d440af7a0 con 0x7f4d4409ac40 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284373699 0 0) 0x7f4d440ae720 con 0x7f4d44136a40 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d440ae200 con 0x7f4d44136a40 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 64806164 0 0) 0x7f4d30004eb0 con 0x7f4d440adbe0 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d440ae720 con 0x7f4d440adbe0 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d340017e0 con 0x7f4d440adbe0 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142701844 0 0) 0x7f4d440af7a0 con 0x7f4d4409ac40 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d30004eb0 con 0x7f4d4409ac40 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d38001b80 con 0x7f4d4409ac40 2024-03-29T11:08:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2064604408 0 0) 0x7f4d440ae200 con 0x7f4d44136a40 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d440af7a0 con 0x7f4d44136a40 2024-03-29T11:08:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d3c001b50 con 0x7f4d44136a40 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1373555481 0 0) 0x7f4d440ae720 con 0x7f4d440adbe0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 >> v1:172.21.15.105:6790/0 conn(0x7f4d4409ac40 legacy=0x7f4d440ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 >> v1:172.21.15.105:6789/0 conn(0x7f4d44136a40 legacy=0x7f4d440b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 monclient: found mon.b 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440b6c20 con 0x7f4d440adbe0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient: _renew_subs 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d440b6ef0 con 0x7f4d440adbe0 2024-03-29T11:08:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient: _renew_subs 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d441e3d10 con 0x7f4d440adbe0 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d49b32640 1 librados: init done 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d34001d50 con 0x7f4d440adbe0 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4d340037b0 con 0x7f4d440adbe0 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 monclient: dump: 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d41ffb640 4 set_mon_vals no callback set 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4d34003d20 con 0x7f4d440adbe0 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2094916495 0 0) 0x7f4d34006700 con 0x7f4d440adbe0 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 10 client.4156.objecter ms_dispatch 0x7f4d44000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.946+0000 7f4d2bfff640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.947+0000 7f4d2bfff640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.947+0000 7f4d49b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.947+0000 7f4d49b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.947+0000 7f4d49b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:30.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.947+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d440ae040 con 0x7f4d440adbe0 2024-03-29T11:08:30.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.949+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4d340042b0 con 0x7f4d440adbe0 2024-03-29T11:08:30.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.949+0000 7f4d2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:30.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:30.949+0000 7f4d2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:30.999 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.020+0000 7fd612032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:08:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.022+0000 7f4d49b32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f231f5e6-d6ae-48a0-8a56-52f22a494330", "id": 6}] 2024-03-29T11:08:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.022+0000 7f4d49b32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f231f5e6-d6ae-48a0-8a56-52f22a494330", "id": 6}] 2024-03-29T11:08:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.022+0000 7f4d49b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.022+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f231f5e6-d6ae-48a0-8a56-52f22a494330", "id": 6} v 0) v1 -- 0x7f4d440ae390 con 0x7f4d440adbe0 2024-03-29T11:08:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d2bfff640 1 -- 172.21.15.110:0/3206498037 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f231f5e6-d6ae-48a0-8a56-52f22a494330", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 285813753 0 1113811849) 0x7f4d340057c0 con 0x7f4d440adbe0 2024-03-29T11:08:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f231f5e6-d6ae-48a0-8a56-52f22a494330", "id": 6}] 2024-03-29T11:08:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:31.004 INFO:teuthology.orchestra.run.smithi110.stdout:6 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 10 librados: watch_flush enter 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 10 librados: watch_flush exit 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 10 monclient: shutdown 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.025+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 >> v1:172.21.15.110:6789/0 conn(0x7f4d440adbe0 legacy=0x7f4d44136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.026+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 shutdown_connections 2024-03-29T11:08:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.026+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 >> 172.21.15.110:0/3206498037 conn(0x7f4d44066b00 msgr2=0x7f4d4413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.026+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 shutdown_connections 2024-03-29T11:08:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.026+0000 7f4d49b32640 1 -- 172.21.15.110:0/3206498037 wait complete. 2024-03-29T11:08:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.026+0000 7f4d49b32640 1 librados: shutdown 2024-03-29T11:08:31.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 7b03c3d9-2138-4e1b-8a2e-382bd235ef36 7 2024-03-29T11:08:31.020 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.041+0000 7f8921e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.136+0000 7f191bc93640 10 monclient: get_monmap_and_config 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.136+0000 7f191bc93640 10 monclient: build_initial_monmap 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.136+0000 7f191bc93640 10 monclient: monmap: 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:08:31.137940+0000 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:08:31.137940+0000 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 1 Processor -- start 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 1 -- start start 2024-03-29T11:08:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient: init 2024-03-29T11:08:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-a con 0x7f191409ac40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-b con 0x7f191409c870 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-c con 0x7f19140b4360 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191409dec0 con 0x7f191409ac40 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191409da30 con 0x7f19140b4360 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140b39c0 con 0x7f191409c870 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.137+0000 7f191bc93640 10 monclient(hunting): authenticate will time out at 791.329081s 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191a230640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f19140b4360 0x7f191414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56752/0 (socket says 172.21.15.110:56752) 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191a230640 1 -- 172.21.15.110:0/939195841 learned_addr learned my addr 172.21.15.110:0/939195841 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7661933 0 0) 0x7f191409da30 con 0x7f19140b4360 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1900003520 con 0x7f19140b4360 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873825598 0 0) 0x7f19140b39c0 con 0x7f191409c870 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191409da30 con 0x7f191409c870 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282001747 0 0) 0x7f1910001110 con 0x7f191409ac40 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:08:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19140b39c0 con 0x7f191409ac40 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041153284 0 0) 0x7f1900003520 con 0x7f19140b4360 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 >> v1:172.21.15.105:6790/0 conn(0x7f191409c870 legacy=0x7f191413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 >> v1:172.21.15.105:6789/0 conn(0x7f191409ac40 legacy=0x7f19140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: _finish_hunting 0 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 monclient: found mon.noname-c 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140b5160 con 0x7f19140b4360 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: _finish_auth 0 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: _check_auth_tickets 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f190c001a10 con 0x7f19140b4360 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191bc93640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: dump: 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191bc93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c001bf0 con 0x7f19140b4360 2024-03-29T11:08:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191bc93640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 1 -- 172.21.15.110:0/939195841 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f190c002000 con 0x7f19140b4360 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f191bc93640 4 set_mon_vals no callback set 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.138+0000 7f1918a2d640 10 monclient: dump: 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 10 monclient: shutdown 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 1 -- 172.21.15.110:0/939195841 >> v1:172.21.15.110:6789/0 conn(0x7f19140b4360 legacy=0x7f191414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 1 -- 172.21.15.110:0/939195841 shutdown_connections 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 1 -- 172.21.15.110:0/939195841 >> 172.21.15.110:0/939195841 conn(0x7f1914066b00 msgr2=0x7f1914066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 1 -- 172.21.15.110:0/939195841 shutdown_connections 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 1 -- 172.21.15.110:0/939195841 wait complete. 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 10 monclient: build_initial_monmap 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.139+0000 7f191bc93640 10 monclient: monmap: 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 librados: starting msgr at 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 librados: starting objecter 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 Processor -- start 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 -- start start 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 librados: setting wanted keys 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 librados: calling monclient init 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient: init 2024-03-29T11:08:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-a con 0x7f191409ac40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-c con 0x7f19141d7bd0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient(hunting): picked mon.noname-b con 0x7f1914140600 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914136aa0 con 0x7f191409ac40 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19141423a0 con 0x7f1914140600 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19141d76d0 con 0x7f19141d7bd0 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191bc93640 10 monclient(hunting): authenticate will time out at 791.331920s 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191a230640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1914140600 0x7f1914140a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56756/0 (socket says 172.21.15.110:56756) 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191a230640 1 -- 172.21.15.110:0/3145130874 learned_addr learned my addr 172.21.15.110:0/3145130874 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f1919a2f640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f191409ac40 0x7f1914142b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.110:36630/0 (socket says 172.21.15.110:36630) 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f191922e640 1 --1- 172.21.15.110:0/3145130874 >> v1:172.21.15.105:6790/0 conn(0x7f19141d7bd0 0x7f19141d8060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.110:51366/0 (socket says 172.21.15.110:51366) 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f190a7fc640 10 client.?.objecter ms_handle_connect 0x7f1914140600 2024-03-29T11:08:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.140+0000 7f190a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 client.?.objecter ms_handle_connect 0x7f19141d7bd0 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 client.?.objecter ms_handle_connect 0x7f191409ac40 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093874684 0 0) 0x7f19141423a0 con 0x7f1914140600 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): my global_id is 4162 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8004eb0 con 0x7f1914140600 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768485617 0 0) 0x7f19141d76d0 con 0x7f19141d7bd0 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): my global_id is 4163 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19141423a0 con 0x7f19141d7bd0 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846123040 0 0) 0x7f1914136aa0 con 0x7f191409ac40 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19141d76d0 con 0x7f191409ac40 2024-03-29T11:08:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141270149 0 0) 0x7f18f8004eb0 con 0x7f1914140600 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1914136aa0 con 0x7f1914140600 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f190c001fc0 con 0x7f1914140600 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): dump: 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091861232 0 0) 0x7f19141423a0 con 0x7f19141d7bd0 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18f8004eb0 con 0x7f19141d7bd0 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f19040018b0 con 0x7f19141d7bd0 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): dump: 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2291120745 0 0) 0x7f19141d76d0 con 0x7f191409ac40 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19141423a0 con 0x7f191409ac40 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1910001820 con 0x7f191409ac40 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient(hunting): dump: 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3438451521 0 0) 0x7f1914136aa0 con 0x7f1914140600 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 >> v1:172.21.15.105:6790/0 conn(0x7f19141d7bd0 legacy=0x7f19141d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 >> v1:172.21.15.105:6789/0 conn(0x7f191409ac40 legacy=0x7f1914142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 monclient: found mon.b 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140b1a80 con 0x7f1914140600 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 5 monclient: authenticate success, global_id 4162 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 10 monclient: _renew_subs 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191413ffa0 con 0x7f1914140600 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 10 monclient: _renew_subs 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19141e3f20 con 0x7f1914140600 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f191bc93640 1 librados: init done 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c0014f0 con 0x7f1914140600 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f190c004fb0 con 0x7f1914140600 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 monclient: dump: 2024-03-29T11:08:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190bfff640 4 set_mon_vals no callback set 2024-03-29T11:08:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f190c0054e0 con 0x7f1914140600 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2619967799 0 0) 0x7f190c006d70 con 0x7f1914140600 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 10 client.4162.objecter ms_dispatch 0x7f1914000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.141+0000 7f190a7fc640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.142+0000 7f191bc93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.142+0000 7f191bc93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.142+0000 7f191bc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.142+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc005080 con 0x7f1914140600 2024-03-29T11:08:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.144+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f190c0067c0 con 0x7f1914140600 2024-03-29T11:08:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.144+0000 7f190a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.144+0000 7f190a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:31.144 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.153+0000 7fd612032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:08:31.161 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.182+0000 7f8921e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:08:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.217+0000 7f191bc93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7b03c3d9-2138-4e1b-8a2e-382bd235ef36", "id": 7}] 2024-03-29T11:08:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.217+0000 7f191bc93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7b03c3d9-2138-4e1b-8a2e-382bd235ef36", "id": 7}] 2024-03-29T11:08:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.217+0000 7f191bc93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.217+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7b03c3d9-2138-4e1b-8a2e-382bd235ef36", "id": 7} v 0) v1 -- 0x7f18fc0058b0 con 0x7f1914140600 2024-03-29T11:08:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.220+0000 7f190a7fc640 1 -- 172.21.15.110:0/3145130874 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b03c3d9-2138-4e1b-8a2e-382bd235ef36", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3175679441 0 1371665406) 0x7f190c006a30 con 0x7f1914140600 2024-03-29T11:08:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.220+0000 7f190a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7b03c3d9-2138-4e1b-8a2e-382bd235ef36", "id": 7}] 2024-03-29T11:08:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.220+0000 7f190a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:31.199 INFO:teuthology.orchestra.run.smithi110.stdout:7 2024-03-29T11:08:31.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 10 librados: watch_flush enter 2024-03-29T11:08:31.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 10 librados: watch_flush exit 2024-03-29T11:08:31.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-29T11:08:31.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 10 client.4162.objecter successfully canceled tick 2024-03-29T11:08:31.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 10 monclient: shutdown 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 >> v1:172.21.15.110:6789/0 conn(0x7f1914140600 legacy=0x7f1914140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 shutdown_connections 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 >> 172.21.15.110:0/3145130874 conn(0x7f1914066b00 msgr2=0x7f19140b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 shutdown_connections 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 -- 172.21.15.110:0/3145130874 wait complete. 2024-03-29T11:08:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-29T11:08:31.221+0000 7f191bc93640 1 librados: shutdown 2024-03-29T11:08:31.210 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:08:31.210 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:08:31.213 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:08:31.213 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:08:31.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:08:31.215 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:08:31.215 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:08:31.215 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:08:31.217 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:08:31.218 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:08:31.218 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:08:31.220 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:08:31.220 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:08:31.220 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:08:31.248 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.270+0000 7fd612032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:08:31.253 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:08:31.253 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:08:31.253 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:08:31.255 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:08:31.255 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:08:31.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:08:31.258 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:08:31.258 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:08:31.258 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:08:31.260 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:08:31.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:31.280 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.301+0000 7f8921e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:08:31.311 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.332+0000 7fd612032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:08:31.334 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.355+0000 7f8921e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 10 monclient: get_monmap_and_config 2024-03-29T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 10 monclient: build_initial_monmap 2024-03-29T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 10 monclient: monmap: 2024-03-29T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 0 2024-03-29T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:08:31.400315+0000 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:08:31.400315+0000 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 1 Processor -- start 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 1 -- start start 2024-03-29T11:08:31.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.399+0000 7f550515d640 10 monclient: init 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient(hunting): picked mon.noname-c con 0x7f550009ac40 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient(hunting): picked mon.noname-b con 0x7f550009c870 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient(hunting): picked mon.noname-a con 0x7f55000b4360 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550009dec0 con 0x7f55000b4360 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550009da30 con 0x7f550009ac40 2024-03-29T11:08:31.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55000b39c0 con 0x7f550009c870 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f550515d640 10 monclient(hunting): authenticate will time out at 785.676563s 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fe59c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f550009c870 0x7f550013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:35038/0 (socket says 172.21.15.105:35038) 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fe59c640 1 -- 172.21.15.105:0/2544580688 learned_addr learned my addr 172.21.15.105:0/2544580688 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54ff59e640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f55000b4360 0x7f550014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46322/0 (socket says 172.21.15.105:46322) 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586163372 0 0) 0x7f54f8001490 con 0x7f55000b4360 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e8003520 con 0x7f55000b4360 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216096651 0 0) 0x7f55000b39c0 con 0x7f550009c870 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f8001490 con 0x7f550009c870 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180075150 0 0) 0x7f550009da30 con 0x7f550009ac40 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.400+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55000b39c0 con 0x7f550009ac40 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785031873 0 0) 0x7f54f8001490 con 0x7f550009c870 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 >> v1:172.21.15.110:6789/0 conn(0x7f550009ac40 legacy=0x7f55000b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 >> v1:172.21.15.105:6789/0 conn(0x7f55000b4360 legacy=0x7f550014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 monclient: found mon.noname-b 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55000b5160 con 0x7f550009c870 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:08:31.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f0001670 con 0x7f550009c870 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f550515d640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f550515d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: dump: 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.393 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f550515d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:31.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54f00017d0 con 0x7f550009c870 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 1 -- 172.21.15.105:0/2544580688 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f0001c20 con 0x7f550009c870 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f550515d640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f550515d640 4 set_mon_vals no callback set 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.401+0000 7f54fdd9b640 10 monclient: dump: 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.395 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 10 monclient: shutdown 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:31.396 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 1 -- 172.21.15.105:0/2544580688 >> v1:172.21.15.105:6790/0 conn(0x7f550009c870 legacy=0x7f550013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 1 -- 172.21.15.105:0/2544580688 shutdown_connections 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 1 -- 172.21.15.105:0/2544580688 >> 172.21.15.105:0/2544580688 conn(0x7f5500066b00 msgr2=0x7f5500066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 1 -- 172.21.15.105:0/2544580688 shutdown_connections 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.402+0000 7f550515d640 1 -- 172.21.15.105:0/2544580688 wait complete. 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 10 monclient: build_initial_monmap 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 10 monclient: monmap: 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 0 2024-03-29T11:08:31.397 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:created 0.000000 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 1 librados: starting msgr at 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 1 librados: starting objecter 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 1 Processor -- start 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 1 -- start start 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.403+0000 7f550515d640 1 librados: setting wanted keys 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 1 librados: calling monclient init 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient: init 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient(hunting): picked mon.noname-a con 0x7f550009ac40 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient(hunting): picked mon.noname-b con 0x7f5500139af0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient(hunting): picked mon.noname-c con 0x7f55001da3a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55000b1670 con 0x7f550009ac40 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55000b18e0 con 0x7f5500139af0 2024-03-29T11:08:31.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55000b0b80 con 0x7f55001da3a0 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f550515d640 10 monclient(hunting): authenticate will time out at 785.680864s 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f54fed9d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f550009ac40 0x7f55000acfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46330/0 (socket says 172.21.15.105:46330) 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.404+0000 7f54fed9d640 1 -- 172.21.15.105:0/504934479 learned_addr learned my addr 172.21.15.105:0/504934479 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter ms_handle_connect 0x7f550009ac40 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter ms_handle_connect 0x7f55001da3a0 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter ms_handle_connect 0x7f5500139af0 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651972161 0 0) 0x7f55000b1670 con 0x7f550009ac40 2024-03-29T11:08:31.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ec004eb0 con 0x7f550009ac40 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726003456 0 0) 0x7f55000b0b80 con 0x7f55001da3a0 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55000b1670 con 0x7f55001da3a0 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679869100 0 0) 0x7f55000b18e0 con 0x7f5500139af0 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:31.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55000b0b80 con 0x7f5500139af0 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831434622 0 0) 0x7f54ec004eb0 con 0x7f550009ac40 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55000b18e0 con 0x7f550009ac40 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f40019d0 con 0x7f550009ac40 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): dump: 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.402 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928886348 0 0) 0x7f55000b1670 con 0x7f55001da3a0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ec004eb0 con 0x7f55001da3a0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4191889805 0 0) 0x7f55000b0b80 con 0x7f5500139af0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55000b1670 con 0x7f5500139af0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f8001ab0 con 0x7f55001da3a0 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): dump: 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.403 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f0001630 con 0x7f5500139af0 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient(hunting): dump: 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.404 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2565949916 0 0) 0x7f55000b18e0 con 0x7f550009ac40 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 >> v1:172.21.15.105:6790/0 conn(0x7f55001da3a0 legacy=0x7f55000aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 >> v1:172.21.15.110:6789/0 conn(0x7f5500139af0 legacy=0x7f55000af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 monclient: found mon.a 2024-03-29T11:08:31.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55000b0660 con 0x7f550009ac40 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 10 monclient: _renew_subs 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55000b08b0 con 0x7f550009ac40 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 10 monclient: _renew_subs 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001dad20 con 0x7f550009ac40 2024-03-29T11:08:31.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f550515d640 1 librados: init done 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54f4001c50 con 0x7f550009ac40 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f54f4003a40 con 0x7f550009ac40 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54fd59a640 4 set_mon_vals no callback set 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 monclient: dump: 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:epoch 1 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:31.407 INFO:teuthology.orchestra.run.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f54f4003fb0 con 0x7f550009ac40 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f54f40069d0 con 0x7f550009ac40 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 10 client.4185.objecter ms_dispatch 0x7f5500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.405+0000 7f54e77fe640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.406+0000 7f54e77fe640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.406+0000 7f550515d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.406+0000 7f550515d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.406+0000 7f550515d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.406+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e0005080 con 0x7f550009ac40 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.408+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f54f4006300 con 0x7f550009ac40 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.408+0000 7f54e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:31.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.408+0000 7f54e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:31.432 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.453+0000 7fd612032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:08:31.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.469+0000 7f550515d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:31.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.469+0000 7f550515d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:31.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.469+0000 7f550515d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:31.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.469+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54e0005330 con 0x7f550009ac40 2024-03-29T11:08:31.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f54e77fe640 1 -- 172.21.15.105:0/504934479 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f54f4006560 con 0x7f550009ac40 2024-03-29T11:08:31.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f54e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:31.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f54e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:31.449 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-29T11:08:31.449 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:31.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f550515d640 10 librados: watch_flush enter 2024-03-29T11:08:31.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f550515d640 10 librados: watch_flush exit 2024-03-29T11:08:31.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f550515d640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-29T11:08:31.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.470+0000 7f550515d640 10 client.4185.objecter successfully canceled tick 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 10 monclient: shutdown 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 >> v1:172.21.15.105:6789/0 conn(0x7f550009ac40 legacy=0x7f55000acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 shutdown_connections 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 >> 172.21.15.105:0/504934479 conn(0x7f5500066b00 msgr2=0x7f550013afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 shutdown_connections 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 -- 172.21.15.105:0/504934479 wait complete. 2024-03-29T11:08:31.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-29T11:08:31.471+0000 7f550515d640 1 librados: shutdown 2024-03-29T11:08:31.451 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.470+0000 7f8921e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:08:31.460 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:08:31.460 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:08:31.531 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.552+0000 7fd612032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:08:31.542 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.563+0000 7f8921e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:08:31.686 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.707+0000 7fd612032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:08:31.699 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.720+0000 7f8921e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:08:31.728 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.749+0000 7fd612032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:08:31.740 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.762+0000 7f8921e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:08:31.823 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-03-29T11:08:31.844+0000 7fce3e6d7640 -1 Falling back to public interface 2024-03-29T11:08:31.823 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-03-29T11:08:31.845+0000 7f4f70e13640 -1 Falling back to public interface 2024-03-29T11:08:31.824 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-03-29T11:08:31.845+0000 7fce3d686640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6803/0: (98) Address already in use 2024-03-29T11:08:31.829 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-03-29T11:08:31.850+0000 7fe317b93640 -1 Falling back to public interface 2024-03-29T11:08:31.834 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-03-29T11:08:31.855+0000 7f1cdf889640 -1 Falling back to public interface 2024-03-29T11:08:31.879 INFO:tasks.ceph.osd.4.smithi110.stderr:2024-03-29T11:08:31.900+0000 7f0e39c1c640 -1 Falling back to public interface 2024-03-29T11:08:31.880 INFO:tasks.ceph.osd.6.smithi110.stderr:2024-03-29T11:08:31.901+0000 7f08e5e12640 -1 Falling back to public interface 2024-03-29T11:08:31.882 INFO:tasks.ceph.osd.7.smithi110.stderr:2024-03-29T11:08:31.903+0000 7fbd36a17640 -1 Falling back to public interface 2024-03-29T11:08:31.885 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-03-29T11:08:31.906+0000 7f2509ddf640 -1 Falling back to public interface 2024-03-29T11:08:31.885 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:31.907+0000 7fd612032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:08:31.897 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:31.919+0000 7f8921e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:08:32.265 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.377+0000 7f0d9136d640 10 monclient: get_monmap_and_config 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.377+0000 7f0d9136d640 10 monclient: build_initial_monmap 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.377+0000 7f0d9136d640 10 monclient: monmap: 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:08:32.378437+0000 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:08:32.378437+0000 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:32.356 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:32.357 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 1 Processor -- start 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 1 -- start start 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient: init 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8c1457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8c13f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8c0a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0aaef0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c145bd0 con 0x7f0d8c13f650 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c145eb0 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:32.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d9136d640 10 monclient(hunting): authenticate will time out at 786.654734s 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d8affd640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f0d8c1457f0 0x7f0d8c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46336/0 (socket says 172.21.15.105:46336) 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d8affd640 1 -- 172.21.15.105:0/1489409273 learned_addr learned my addr 172.21.15.105:0/1489409273 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.378+0000 7f0d8b7fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f0d8c0a9d70 0x7f0d8c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:35052/0 (socket says 172.21.15.105:35052) 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358470510 0 0) 0x7f0d8c145eb0 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d78003520 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020182924 0 0) 0x7f0d8c0aaef0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:08:32.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c145eb0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130809979 0 0) 0x7f0d8c145bd0 con 0x7f0d8c13f650 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c0aaef0 con 0x7f0d8c13f650 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101359150 0 0) 0x7f0d78003520 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 >> v1:172.21.15.110:6789/0 conn(0x7f0d8c13f650 legacy=0x7f0d8c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 >> v1:172.21.15.105:6789/0 conn(0x7f0d8c1457f0 legacy=0x7f0d8c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c146340 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d84001a10 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: dump: 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.369 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d9136d640 5 monclient: authenticate success, global_id 4196 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d9136d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d84001b70 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 1 -- 172.21.15.105:0/1489409273 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d84001fc0 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d9136d640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d9136d640 4 set_mon_vals no callback set 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.379+0000 7f0d8a7fc640 10 monclient: dump: 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.370 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 10 monclient: shutdown 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 -- 172.21.15.105:0/1489409273 >> v1:172.21.15.105:6790/0 conn(0x7f0d8c0a9d70 legacy=0x7f0d8c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 -- 172.21.15.105:0/1489409273 shutdown_connections 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 -- 172.21.15.105:0/1489409273 >> 172.21.15.105:0/1489409273 conn(0x7f0d8c066a00 msgr2=0x7f0d8c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 -- 172.21.15.105:0/1489409273 shutdown_connections 2024-03-29T11:08:32.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 -- 172.21.15.105:0/1489409273 wait complete. 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 10 monclient: build_initial_monmap 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 10 monclient: monmap: 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 librados: starting msgr at 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.380+0000 7f0d9136d640 1 librados: starting objecter 2024-03-29T11:08:32.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 Processor -- start 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 -- start start 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 librados: setting wanted keys 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 librados: calling monclient init 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient: init 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8c0a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8c13f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8c1457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0adf20 con 0x7f0d8c1457f0 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c141ec0 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c142050 con 0x7f0d8c13f650 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d9136d640 10 monclient(hunting): authenticate will time out at 786.657638s 2024-03-29T11:08:32.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d8b7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f0d8c1457f0 0x7f0d8c1d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46338/0 (socket says 172.21.15.105:46338) 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d8b7fe640 1 -- 172.21.15.105:0/2478143835 learned_addr learned my addr 172.21.15.105:0/2478143835 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d83fff640 1 --1- 172.21.15.105:0/2478143835 >> v1:172.21.15.105:6790/0 conn(0x7f0d8c13f650 0x7f0d8c1d7110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:35064/0 (socket says 172.21.15.105:35064) 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter ms_handle_connect 0x7f0d8c1457f0 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter ms_handle_connect 0x7f0d8c13f650 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter ms_handle_connect 0x7f0d8c0a9d70 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888400111 0 0) 0x7f0d8c0adf20 con 0x7f0d8c1457f0 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c004eb0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435020405 0 0) 0x7f0d8c142050 con 0x7f0d8c13f650 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): my global_id is 4199 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c0adf20 con 0x7f0d8c13f650 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306576772 0 0) 0x7f0d8c141ec0 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 10 monclient(hunting): my global_id is 4198 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.381+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c142050 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715073562 0 0) 0x7f0d6c004eb0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d8c141ec0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575961832 0 0) 0x7f0d8c0adf20 con 0x7f0d8c13f650 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d6c004eb0 con 0x7f0d8c13f650 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d840019d0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): dump: 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.376 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108591915 0 0) 0x7f0d8c142050 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d8c0adf20 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d740018f0 con 0x7f0d8c13f650 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): dump: 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.377 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d7c001770 con 0x7f0d8c0a9d70 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient(hunting): dump: 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.378 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384818210 0 0) 0x7f0d8c141ec0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 >> v1:172.21.15.105:6790/0 conn(0x7f0d8c13f650 legacy=0x7f0d8c1d7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 >> v1:172.21.15.110:6789/0 conn(0x7f0d8c0a9d70 legacy=0x7f0d8c139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 monclient: found mon.a 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c1421e0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: _finish_auth 0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 10 monclient: _renew_subs 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:32.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c13a490 con 0x7f0d8c1457f0 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 10 monclient: _renew_subs 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c0ad150 con 0x7f0d8c1457f0 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d9136d640 1 librados: init done 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d840014d0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d84004fb0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d89ffb640 4 set_mon_vals no callback set 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 monclient: dump: 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:32.380 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0d84005520 con 0x7f0d8c1457f0 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f0d840067a0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 10 client.4191.objecter ms_dispatch 0x7f0d8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:32.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.382+0000 7f0d837fe640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.383+0000 7f0d9136d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.383+0000 7f0d9136d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.383+0000 7f0d9136d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.383+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d70005080 con 0x7f0d8c1457f0 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.385+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0d840069d0 con 0x7f0d8c1457f0 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.385+0000 7f0d837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:32.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.385+0000 7f0d837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:32.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.445+0000 7f0d9136d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:32.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.445+0000 7f0d9136d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:32.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.445+0000 7f0d9136d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:32.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.445+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d70005330 con 0x7f0d8c1457f0 2024-03-29T11:08:32.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d837fe640 1 -- 172.21.15.105:0/2478143835 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f0d84005880 con 0x7f0d8c1457f0 2024-03-29T11:08:32.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:32.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:32.425 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:08:32.425 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:32.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 10 librados: watch_flush enter 2024-03-29T11:08:32.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 10 librados: watch_flush exit 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 10 monclient: shutdown 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 >> v1:172.21.15.105:6789/0 conn(0x7f0d8c1457f0 legacy=0x7f0d8c1d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 shutdown_connections 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 >> 172.21.15.105:0/2478143835 conn(0x7f0d8c066a00 msgr2=0x7f0d8c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.446+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 shutdown_connections 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.447+0000 7f0d9136d640 1 -- 172.21.15.105:0/2478143835 wait complete. 2024-03-29T11:08:32.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:32.447+0000 7f0d9136d640 1 librados: shutdown 2024-03-29T11:08:32.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:32.691 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:32.712+0000 7f8921e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:08:32.704 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:32.726+0000 7fd612032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:08:32.964 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:32.985+0000 7f8921e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:08:32.978 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:33.000+0000 7fd612032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:08:33.017 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:33.038+0000 7f8921e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:08:33.029 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:33.050+0000 7fd612032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:08:33.130 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:33.152+0000 7f8921e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:08:33.144 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-03-29T11:08:33.166+0000 7fd612032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:08:34.144 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:34.165+0000 7f88c23ae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:08:34.144 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-03-29T11:08:34.165+0000 7f88c23ae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:08:35.289 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-03-29T11:08:35.310+0000 7fce3e6d7640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:08:35.336 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-03-29T11:08:35.357+0000 7f4f70e13640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:08:35.336 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-03-29T11:08:35.357+0000 7fe317b93640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:08:35.342 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-03-29T11:08:35.364+0000 7f1cdf889640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:08:35.367 INFO:tasks.ceph.osd.4.smithi110.stderr:2024-03-29T11:08:35.385+0000 7f0e39c1c640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:08:35.412 INFO:tasks.ceph.osd.6.smithi110.stderr:2024-03-29T11:08:35.433+0000 7f08e5e12640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:08:35.421 INFO:tasks.ceph.osd.7.smithi110.stderr:2024-03-29T11:08:35.441+0000 7fbd36a17640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:08:35.425 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-03-29T11:08:35.445+0000 7f2509ddf640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:08:39.241 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.427+0000 7f19f44e2640 10 monclient: get_monmap_and_config 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.427+0000 7f19f44e2640 10 monclient: build_initial_monmap 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.427+0000 7f19f44e2640 10 monclient: monmap: 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:08:39.428281+0000 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:08:39.428281+0000 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.406 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.428+0000 7f19f44e2640 1 Processor -- start 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.428+0000 7f19f44e2640 1 -- start start 2024-03-29T11:08:39.407 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.428+0000 7f19f44e2640 10 monclient: init 2024-03-29T11:08:39.408 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f44e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:39.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f44e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:39.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-b con 0x7f19ec1457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:39.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-a con 0x7f19ec13f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:39.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f227e640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f19ec1457f0 0x7f19ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46370/0 (socket says 172.21.15.105:46370) 2024-03-29T11:08:39.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.430+0000 7f19f227e640 1 -- 172.21.15.105:0/102733683 learned_addr learned my addr 172.21.15.105:0/102733683 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-c con 0x7f19ec0a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0aaef0 con 0x7f19ec13f650 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec145bd0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec145eb0 con 0x7f19ec1457f0 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f44e2640 10 monclient(hunting): authenticate will time out at 793.707089s 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981634589 0 0) 0x7f19ec0aaef0 con 0x7f19ec13f650 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _negotiate 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:08:39.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8003520 con 0x7f19ec13f650 2024-03-29T11:08:39.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890479391 0 0) 0x7f19ec145bd0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _negotiate 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.431+0000 7f19f127c640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ec0aaef0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413141124 0 0) 0x7f19ec145eb0 con 0x7f19ec1457f0 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient(hunting): _negotiate 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ec145bd0 con 0x7f19ec1457f0 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984881720 0 0) 0x7f19d8003520 con 0x7f19ec13f650 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 >> v1:172.21.15.105:6790/0 conn(0x7f19ec1457f0 legacy=0x7f19ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 >> v1:172.21.15.110:6789/0 conn(0x7f19ec0a9d70 legacy=0x7f19ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient: _finish_hunting 0 2024-03-29T11:08:39.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 monclient: found mon.noname-a 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec146340 con 0x7f19ec13f650 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient: _finish_auth 0 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient: _check_auth_tickets 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f44e2640 5 monclient: authenticate success, global_id 4236 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f44e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f19e80015c0 con 0x7f19ec13f650 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.432+0000 7f19f127c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: dump: 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.413 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f44e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e80017e0 con 0x7f19ec13f650 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 1 -- 172.21.15.105:0/102733683 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f19e8001b70 con 0x7f19ec13f650 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f127c640 10 monclient: dump: 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:39.414 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f44e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:39.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.433+0000 7f19f44e2640 4 set_mon_vals no callback set 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.434+0000 7f19f44e2640 10 monclient: shutdown 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.434+0000 7f19f44e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.434+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 >> v1:172.21.15.105:6789/0 conn(0x7f19ec13f650 legacy=0x7f19ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.434+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 shutdown_connections 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.434+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 >> 172.21.15.105:0/102733683 conn(0x7f19ec066a00 msgr2=0x7f19ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.435+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 shutdown_connections 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.435+0000 7f19f44e2640 1 -- 172.21.15.105:0/102733683 wait complete. 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.436+0000 7f19f44e2640 10 monclient: build_initial_monmap 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.436+0000 7f19f44e2640 10 monclient: monmap: 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:39.421 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 librados: starting msgr at 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 librados: starting objecter 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 Processor -- start 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 -- start start 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 librados: setting wanted keys 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 1 librados: calling monclient init 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.437+0000 7f19f44e2640 10 monclient: init 2024-03-29T11:08:39.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.438+0000 7f19f44e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.438+0000 7f19f44e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.438+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-b con 0x7f19ec0a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-c con 0x7f19ec13f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f1a7d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f19ec13f650 0x7f19ec0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46554/0 (socket says 172.21.15.105:46554) 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f1a7d640 1 -- 172.21.15.105:0/1906733740 learned_addr learned my addr 172.21.15.105:0/1906733740 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 10 monclient(hunting): picked mon.noname-a con 0x7f19ec1457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0adce0 con 0x7f19ec1457f0 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0af450 con 0x7f19ec0a9d70 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 client.?.objecter ms_handle_connect 0x7f19ec13f650 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0ad920 con 0x7f19ec13f650 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19f44e2640 10 monclient(hunting): authenticate will time out at 793.715470s 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 client.?.objecter ms_handle_connect 0x7f19ec0a9d70 2024-03-29T11:08:39.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306902824 0 0) 0x7f19ec0af450 con 0x7f19ec0a9d70 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.439+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0004eb0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1101312440 0 0) 0x7f19d0004eb0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19ec0af450 con 0x7f19ec0a9d70 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f19e00017f0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): dump: 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:39.424 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235415051 0 0) 0x7f19ec0ad920 con 0x7f19ec13f650 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0004eb0 con 0x7f19ec13f650 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 client.?.objecter ms_handle_connect 0x7f19ec1457f0 2024-03-29T11:08:39.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3999197809 0 0) 0x7f19ec0af450 con 0x7f19ec0a9d70 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 >> v1:172.21.15.105:6790/0 conn(0x7f19ec13f650 legacy=0x7f19ec0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 >> v1:172.21.15.105:6789/0 conn(0x7f19ec1457f0 legacy=0x7f19ec1e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 monclient: found mon.b 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec1d7890 con 0x7f19ec0a9d70 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient: _finish_auth 0 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19f44e2640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19f44e2640 10 monclient: _renew_subs 2024-03-29T11:08:39.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19f44e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.440+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec0ae170 con 0x7f19ec0a9d70 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19f44e2640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19f44e2640 10 monclient: _renew_subs 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19f44e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec142240 con 0x7f19ec0a9d70 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19f44e2640 1 librados: init done 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e0001ca0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f19e0004980 con 0x7f19ec0a9d70 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.441+0000 7f19deffd640 10 monclient: dump: 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.427 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:39.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.442+0000 7f19f0a7b640 4 set_mon_vals no callback set 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.443+0000 7f19f44e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.443+0000 7f19f44e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.443+0000 7f19f44e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.443+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d4005080 con 0x7f19ec0a9d70 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.444+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f19e0004ae0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f19e0084910 con 0x7f19ec0a9d70 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 10 client.4240.objecter ms_dispatch 0x7f19ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:39.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:39.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:39.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:08:39.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19f1a7d640 10 monclient: get_auth_request con 0x7f19d006dd70 auth_method 0 2024-03-29T11:08:39.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.445+0000 7f19deffd640 10 client.4240.objecter ms_handle_connect 0x7f19d006dd70 2024-03-29T11:08:39.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.452+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19e0050250 con 0x7f19ec0a9d70 2024-03-29T11:08:39.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.452+0000 7f19deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:39.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.452+0000 7f19deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:39.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.841+0000 7f19f44e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:39.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.841+0000 7f19f44e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:39.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.841+0000 7f19f44e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:39.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.841+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d4005330 con 0x7f19ec0a9d70 2024-03-29T11:08:39.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.842+0000 7f19deffd640 1 -- 172.21.15.105:0/1906733740 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f19e0083fc0 con 0x7f19ec0a9d70 2024-03-29T11:08:39.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.842+0000 7f19deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:39.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.842+0000 7f19deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:39.821 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:08:39.821 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:39.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.845+0000 7f19f44e2640 10 librados: watch_flush enter 2024-03-29T11:08:39.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.845+0000 7f19f44e2640 10 librados: watch_flush exit 2024-03-29T11:08:39.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.845+0000 7f19f44e2640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:08:39.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.845+0000 7f19f44e2640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:08:39.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.845+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 >> v1:172.21.15.110:6816/1087970004 conn(0x7f19d006dd70 legacy=0x7f19d0070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 10 monclient: shutdown 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 >> v1:172.21.15.110:6789/0 conn(0x7f19ec0a9d70 legacy=0x7f19ec1d7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 shutdown_connections 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 >> 172.21.15.105:0/1906733740 conn(0x7f19ec066a00 msgr2=0x7f19ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:39.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 shutdown_connections 2024-03-29T11:08:39.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 -- 172.21.15.105:0/1906733740 wait complete. 2024-03-29T11:08:39.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:39.846+0000 7f19f44e2640 1 librados: shutdown 2024-03-29T11:08:39.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:46.646 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:46.796 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.818+0000 7f8469e7d640 10 monclient: get_monmap_and_config 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.818+0000 7f8469e7d640 10 monclient: build_initial_monmap 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.818+0000 7f8469e7d640 10 monclient: monmap: 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:08:46.819313+0000 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:08:46.819313+0000 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:46.797 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.798 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.819+0000 7f8469e7d640 1 Processor -- start 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.819+0000 7f8469e7d640 1 -- start start 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.819+0000 7f8469e7d640 10 monclient: init 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f84640b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f846414a700 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f84640ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84640aba70 con 0x7f84640b0770 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84640aecb0 con 0x7f846414a700 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84640af150 con 0x7f84640ac5a0 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8469e7d640 10 monclient(hunting): authenticate will time out at 801.096527s 2024-03-29T11:08:46.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8463fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f84640ac5a0 0x7f8464151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:43424/0 (socket says 172.21.15.105:43424) 2024-03-29T11:08:46.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.820+0000 7f8463fff640 1 -- 172.21.15.105:0/3453050512 learned_addr learned my addr 172.21.15.105:0/3453050512 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:46.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900626919 0 0) 0x7f84640af150 con 0x7f84640ac5a0 2024-03-29T11:08:46.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.821+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8450003520 con 0x7f84640ac5a0 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.822+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2426327541 0 0) 0x7f8450003520 con 0x7f84640ac5a0 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.822+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 >> v1:172.21.15.110:6789/0 conn(0x7f846414a700 legacy=0x7f84640abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:46.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.822+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 >> v1:172.21.15.105:6789/0 conn(0x7f84640b0770 legacy=0x7f8464149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 1 monclient: found mon.noname-b 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84640ac9d0 con 0x7f84640ac5a0 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f845c001a10 con 0x7f84640ac5a0 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:46.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: dump: 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f8469e7d640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f8469e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f845c001ca0 con 0x7f84640ac5a0 2024-03-29T11:08:46.803 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f84627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:46.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f8469e7d640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:46.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.823+0000 7f8469e7d640 4 set_mon_vals no callback set 2024-03-29T11:08:46.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f84627fc640 1 -- 172.21.15.105:0/3453050512 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f845c002030 con 0x7f84640ac5a0 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f84627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f84627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f84627fc640 10 monclient: dump: 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f8469e7d640 10 monclient: shutdown 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f8469e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f8469e7d640 1 -- 172.21.15.105:0/3453050512 >> v1:172.21.15.105:6790/0 conn(0x7f84640ac5a0 legacy=0x7f8464151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f8469e7d640 1 -- 172.21.15.105:0/3453050512 shutdown_connections 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.824+0000 7f8469e7d640 1 -- 172.21.15.105:0/3453050512 >> 172.21.15.105:0/3453050512 conn(0x7f846406b660 msgr2=0x7f846406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.825+0000 7f8469e7d640 1 -- 172.21.15.105:0/3453050512 shutdown_connections 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.825+0000 7f8469e7d640 1 -- 172.21.15.105:0/3453050512 wait complete. 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.826+0000 7f8469e7d640 10 monclient: build_initial_monmap 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.826+0000 7f8469e7d640 10 monclient: monmap: 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.807 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.826+0000 7f8469e7d640 1 librados: starting msgr at 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.826+0000 7f8469e7d640 1 librados: starting objecter 2024-03-29T11:08:46.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.827+0000 7f8469e7d640 1 Processor -- start 2024-03-29T11:08:46.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.827+0000 7f8469e7d640 1 -- start start 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.827+0000 7f8469e7d640 1 librados: setting wanted keys 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.827+0000 7f8469e7d640 1 librados: calling monclient init 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.827+0000 7f8469e7d640 10 monclient: init 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f84640ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f84640b0770 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f846414a700 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84641239d0 con 0x7f846414a700 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84641263f0 con 0x7f84640ac5a0 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84641284a0 con 0x7f84640b0770 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f8469e7d640 10 monclient(hunting): authenticate will time out at 801.104215s 2024-03-29T11:08:46.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f84637fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f84640ac5a0 0x7f8464144fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:33656/0 (socket says 172.21.15.105:33656) 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f84637fe640 1 -- 172.21.15.105:0/877608893 learned_addr learned my addr 172.21.15.105:0/877608893 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f845bfff640 10 client.?.objecter ms_handle_connect 0x7f84640ac5a0 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f845bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f845bfff640 10 client.?.objecter ms_handle_connect 0x7f846414a700 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.828+0000 7f845bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914761147 0 0) 0x7f84641239d0 con 0x7f846414a700 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8444004eb0 con 0x7f846414a700 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334946512 0 0) 0x7f84641263f0 con 0x7f84640ac5a0 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:46.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84641239d0 con 0x7f84640ac5a0 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 client.?.objecter ms_handle_connect 0x7f84640b0770 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916541935 0 0) 0x7f84641284a0 con 0x7f84640b0770 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.829+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84641263f0 con 0x7f84640b0770 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1575860969 0 0) 0x7f8444004eb0 con 0x7f846414a700 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84641284a0 con 0x7f846414a700 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174547277 0 0) 0x7f84641239d0 con 0x7f84640ac5a0 2024-03-29T11:08:46.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8444004eb0 con 0x7f84640ac5a0 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f845c0018d0 con 0x7f846414a700 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:46.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f84540018a0 con 0x7f84640ac5a0 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.830+0000 7f845bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994543256 0 0) 0x7f84641263f0 con 0x7f84640b0770 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84641239d0 con 0x7f84640b0770 2024-03-29T11:08:46.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f844c0016c0 con 0x7f84640b0770 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient(hunting): dump: 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 631808797 0 0) 0x7f84641284a0 con 0x7f846414a700 2024-03-29T11:08:46.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 >> v1:172.21.15.105:6790/0 conn(0x7f84640b0770 legacy=0x7f8464126880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 >> v1:172.21.15.110:6789/0 conn(0x7f84640ac5a0 legacy=0x7f8464144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 monclient: found mon.a 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84641dc1e0 con 0x7f846414a700 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient: _finish_auth 0 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f845bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 10 monclient: _renew_subs 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8464125e40 con 0x7f846414a700 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 10 client.4242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 10 monclient: _renew_subs 2024-03-29T11:08:46.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84641458c0 con 0x7f846414a700 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.831+0000 7f8469e7d640 1 librados: init done 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.832+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f845c001a30 con 0x7f846414a700 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.832+0000 7f845bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.832+0000 7f8461ffb640 4 set_mon_vals no callback set 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.833+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f845c005110 con 0x7f846414a700 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.833+0000 7f845bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.833+0000 7f845bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.833+0000 7f845bfff640 10 monclient: dump: 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:46.820 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f845c01b0e0 con 0x7f846414a700 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f845c084e10 con 0x7f846414a700 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f845bfff640 10 client.4242.objecter ms_dispatch 0x7f8464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f845bfff640 3 client.4242.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f845bfff640 3 client.4242.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:46.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.834+0000 7f84637fe640 10 monclient: get_auth_request con 0x7f844406e100 auth_method 0 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f845bfff640 20 client.4242.objecter dump_active .. 0 homeless 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f8469e7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f8469e7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f8469e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f845bfff640 10 client.4242.objecter ms_handle_connect 0x7f844406e100 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.835+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448005080 con 0x7f846414a700 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.844+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f845c0506f0 con 0x7f846414a700 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.844+0000 7f845bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:46.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:46.844+0000 7f845bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:47.133 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.154+0000 7f8469e7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:47.133 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.154+0000 7f8469e7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:47.133 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.154+0000 7f8469e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:08:47.133 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.154+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8448005330 con 0x7f846414a700 2024-03-29T11:08:47.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.155+0000 7f845bfff640 1 -- 172.21.15.105:0/877608893 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f845c052620 con 0x7f846414a700 2024-03-29T11:08:47.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.155+0000 7f845bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:47.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.155+0000 7f845bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:47.135 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:08:47.135 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.158+0000 7f8469e7d640 10 librados: watch_flush enter 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 10 librados: watch_flush exit 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 20 client.4242.objecter shutdown clearing up homeless session... 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 10 client.4242.objecter successfully canceled tick 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 >> v1:172.21.15.110:6816/1087970004 conn(0x7f844406e100 legacy=0x7f84440705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 10 monclient: shutdown 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 >> v1:172.21.15.105:6789/0 conn(0x7f846414a700 legacy=0x7f84641e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 shutdown_connections 2024-03-29T11:08:47.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.159+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 >> 172.21.15.105:0/877608893 conn(0x7f846406b660 msgr2=0x7f84640b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:47.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.160+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 shutdown_connections 2024-03-29T11:08:47.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.160+0000 7f8469e7d640 1 -- 172.21.15.105:0/877608893 wait complete. 2024-03-29T11:08:47.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:47.160+0000 7f8469e7d640 1 librados: shutdown 2024-03-29T11:08:47.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:53.960 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:54.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.132+0000 7f6bb7bed640 10 monclient: get_monmap_and_config 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.132+0000 7f6bb7bed640 10 monclient: build_initial_monmap 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.132+0000 7f6bb7bed640 10 monclient: monmap: 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:08:54.133254+0000 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:08:54.133254+0000 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:08:54.111 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.132+0000 7f6bb7bed640 1 Processor -- start 2024-03-29T11:08:54.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.133+0000 7f6bb7bed640 1 -- start start 2024-03-29T11:08:54.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.133+0000 7f6bb7bed640 10 monclient: init 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb01422b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb0142690 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb00b17d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb0142c80 con 0x7f6bb01422b0 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00aae40 con 0x7f6bb00b17d0 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb013fbd0 con 0x7f6bb0142690 2024-03-29T11:08:54.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:54.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb7bed640 10 monclient(hunting): authenticate will time out at 808.410537s 2024-03-29T11:08:54.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb618a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6bb00b17d0 0x7f6bb00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:34584/0 (socket says 172.21.15.105:34584) 2024-03-29T11:08:54.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.134+0000 7f6bb618a640 1 -- 172.21.15.105:0/118934888 learned_addr learned my addr 172.21.15.105:0/118934888 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:54.117 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935623490 0 0) 0x7f6bb00aae40 con 0x7f6bb00b17d0 2024-03-29T11:08:54.117 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _negotiate 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9c003520 con 0x7f6bb00b17d0 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434093611 0 0) 0x7f6bb013fbd0 con 0x7f6bb0142690 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _negotiate 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.138+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb00aae40 con 0x7f6bb0142690 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.139+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415917644 0 0) 0x7f6bb00aae40 con 0x7f6bb0142690 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.139+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 >> v1:172.21.15.110:6789/0 conn(0x7f6bb00b17d0 legacy=0x7f6bb00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.139+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 >> v1:172.21.15.105:6789/0 conn(0x7f6bb01422b0 legacy=0x7f6bb013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:54.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.140+0000 7f6bb4987640 10 monclient: _finish_hunting 0 2024-03-29T11:08:54.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 1 monclient: found mon.noname-b 2024-03-29T11:08:54.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:54.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0151940 con 0x7f6bb0142690 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: _finish_auth 0 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: _check_auth_tickets 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb7bed640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb7bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6bac001520 con 0x7f6bb0142690 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb4987640 10 monclient: dump: 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.122 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.141+0000 7f6bb7bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bac001740 con 0x7f6bb0142690 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 1 -- 172.21.15.105:0/118934888 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6bac001a50 con 0x7f6bb0142690 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb7bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb7bed640 4 set_mon_vals no callback set 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:54.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.142+0000 7f6bb4987640 10 monclient: dump: 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 10 monclient: shutdown 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 1 -- 172.21.15.105:0/118934888 >> v1:172.21.15.105:6790/0 conn(0x7f6bb0142690 legacy=0x7f6bb00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 1 -- 172.21.15.105:0/118934888 shutdown_connections 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 1 -- 172.21.15.105:0/118934888 >> 172.21.15.105:0/118934888 conn(0x7f6bb0066a00 msgr2=0x7f6bb0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:54.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 1 -- 172.21.15.105:0/118934888 shutdown_connections 2024-03-29T11:08:54.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.143+0000 7f6bb7bed640 1 -- 172.21.15.105:0/118934888 wait complete. 2024-03-29T11:08:54.126 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 10 monclient: build_initial_monmap 2024-03-29T11:08:54.126 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 10 monclient: monmap: 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 librados: starting msgr at 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 librados: starting objecter 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 Processor -- start 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 -- start start 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 librados: setting wanted keys 2024-03-29T11:08:54.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 1 librados: calling monclient init 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.144+0000 7f6bb7bed640 10 monclient: init 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb00b17d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb01422b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb0142690 addr v1:172.21.15.110:6789/0 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb0132150 con 0x7f6bb01422b0 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00ace10 con 0x7f6bb0142690 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.146+0000 7f6bb7bed640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00ad0f0 con 0x7f6bb00b17d0 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.147+0000 7f6bb7bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.147+0000 7f6bb7bed640 10 monclient(hunting): authenticate will time out at 808.423074s 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.147+0000 7f6bb618a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6bb0142690 0x7f6bb01302b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:34872/0 (socket says 172.21.15.105:34872) 2024-03-29T11:08:54.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.147+0000 7f6bb618a640 1 -- 172.21.15.105:0/1694355323 learned_addr learned my addr 172.21.15.105:0/1694355323 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.148+0000 7f6ba67fc640 10 client.?.objecter ms_handle_connect 0x7f6bb0142690 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.148+0000 7f6ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301214212 0 0) 0x7f6bb00ace10 con 0x7f6bb0142690 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.149+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94004eb0 con 0x7f6bb0142690 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 942971095 0 0) 0x7f6b94004eb0 con 0x7f6bb0142690 2024-03-29T11:08:54.130 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b94005300 con 0x7f6bb0142690 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6ba8001df0 con 0x7f6bb0142690 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 monclient(hunting): dump: 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 client.?.objecter ms_handle_connect 0x7f6bb00b17d0 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 client.?.objecter ms_handle_connect 0x7f6bb01422b0 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.150+0000 7f6ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381376016 0 0) 0x7f6bb0132150 con 0x7f6bb01422b0 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:08:54.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b940058a0 con 0x7f6bb01422b0 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3628972297 0 0) 0x7f6b94005300 con 0x7f6bb0142690 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 >> v1:172.21.15.105:6790/0 conn(0x7f6bb00b17d0 legacy=0x7f6bb00ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 >> v1:172.21.15.105:6789/0 conn(0x7f6bb01422b0 legacy=0x7f6bb012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 monclient: found mon.b 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00acb30 con 0x7f6bb0142690 2024-03-29T11:08:54.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6ba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 10 monclient: _renew_subs 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb00abec0 con 0x7f6bb0142690 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 10 monclient: _renew_subs 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01e2e70 con 0x7f6bb0142690 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.151+0000 7f6bb7bed640 1 librados: init done 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6b9bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6b9bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:54.137 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6b9bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb012db50 con 0x7f6bb0142690 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ba8002050 con 0x7f6bb0142690 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6ba8001df0 con 0x7f6bb0142690 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba7fff640 4 set_mon_vals no callback set 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.154+0000 7f6ba67fc640 10 monclient: dump: 2024-03-29T11:08:54.138 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.155+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f6ba801acc0 con 0x7f6bb0142690 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f6ba80848e0 con 0x7f6bb0142690 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 10 client.4252.objecter ms_dispatch 0x7f6bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6bb5989640 10 monclient: get_auth_request con 0x7f6b9406e210 auth_method 0 2024-03-29T11:08:54.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T11:08:54.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.156+0000 7f6ba67fc640 10 client.4252.objecter ms_handle_connect 0x7f6b9406e210 2024-03-29T11:08:54.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.162+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ba8050220 con 0x7f6bb0142690 2024-03-29T11:08:54.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.162+0000 7f6ba67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:54.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.162+0000 7f6ba67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:54.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.475+0000 7f6b9bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:54.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.475+0000 7f6b9bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:54.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.475+0000 7f6b9bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:08:54.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.475+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bb012db50 con 0x7f6bb0142690 2024-03-29T11:08:54.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.477+0000 7f6ba67fc640 1 -- 172.21.15.105:0/1694355323 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f6bb012db50 con 0x7f6bb0142690 2024-03-29T11:08:54.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.477+0000 7f6ba67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:54.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.477+0000 7f6ba67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:54.459 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:08:54.459 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 10 librados: watch_flush enter 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 10 librados: watch_flush exit 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 10 client.4252.objecter successfully canceled tick 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 >> v1:172.21.15.110:6816/1087970004 conn(0x7f6b9406e210 legacy=0x7f6b940706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 10 monclient: shutdown 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 >> v1:172.21.15.110:6789/0 conn(0x7f6bb0142690 legacy=0x7f6bb01302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 shutdown_connections 2024-03-29T11:08:54.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.481+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 >> 172.21.15.105:0/1694355323 conn(0x7f6bb0066a00 msgr2=0x7f6bb013c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:54.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.482+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 shutdown_connections 2024-03-29T11:08:54.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.482+0000 7f6b9bfff640 1 -- 172.21.15.105:0/1694355323 wait complete. 2024-03-29T11:08:54.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:08:54.482+0000 7f6b9bfff640 1 librados: shutdown 2024-03-29T11:08:54.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:01.284 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:01.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.465+0000 7f5bda231640 10 monclient: get_monmap_and_config 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.465+0000 7f5bda231640 10 monclient: build_initial_monmap 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.466+0000 7f5bda231640 10 monclient: monmap: 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:01.467054+0000 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:01.467054+0000 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:01.445 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.467+0000 7f5bda231640 1 Processor -- start 2024-03-29T11:09:01.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.467+0000 7f5bda231640 1 -- start start 2024-03-29T11:09:01.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.467+0000 7f5bda231640 10 monclient: init 2024-03-29T11:09:01.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd40b0770 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40ad850 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd413c1b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40ac8e0 con 0x7f5bd413c1b0 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40ad330 con 0x7f5bd40b0770 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40acfe0 con 0x7f5bd40ad850 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.468+0000 7f5bda231640 10 monclient(hunting): authenticate will time out at 815.744162s 2024-03-29T11:09:01.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.470+0000 7f5bd2ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f5bd40ad850 0x7f5bd40aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:43719/0 (socket says 172.21.15.105:43719) 2024-03-29T11:09:01.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.470+0000 7f5bd2ffd640 1 -- 172.21.15.105:0/3445528333 learned_addr learned my addr 172.21.15.105:0/3445528333 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720332308 0 0) 0x7f5bd40ac8e0 con 0x7f5bd413c1b0 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc003520 con 0x7f5bd413c1b0 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.475+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485578764 0 0) 0x7f5bd40acfe0 con 0x7f5bd40ad850 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.476+0000 7f5bd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.476+0000 7f5bd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.476+0000 7f5bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.476+0000 7f5bd27fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:09:01.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.476+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40ac8e0 con 0x7f5bd40ad850 2024-03-29T11:09:01.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804945615 0 0) 0x7f5bbc003520 con 0x7f5bd413c1b0 2024-03-29T11:09:01.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 >> v1:172.21.15.105:6790/0 conn(0x7f5bd40ad850 legacy=0x7f5bd40aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 >> v1:172.21.15.110:6789/0 conn(0x7f5bd40b0770 legacy=0x7f5bd4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 monclient: found mon.noname-a 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40ab1f0 con 0x7f5bd413c1b0 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5bc4001690 con 0x7f5bd413c1b0 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.477+0000 7f5bd27fc640 10 monclient: dump: 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:01.457 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc40018e0 con 0x7f5bd413c1b0 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 1 -- 172.21.15.105:0/3445528333 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5bc4001c70 con 0x7f5bd413c1b0 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:01.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bd27fc640 10 monclient: dump: 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bda231640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bda231640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:01.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.478+0000 7f5bda231640 4 set_mon_vals no callback set 2024-03-29T11:09:01.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.479+0000 7f5bda231640 10 monclient: shutdown 2024-03-29T11:09:01.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.479+0000 7f5bda231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:01.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.479+0000 7f5bda231640 1 -- 172.21.15.105:0/3445528333 >> v1:172.21.15.105:6789/0 conn(0x7f5bd413c1b0 legacy=0x7f5bd414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.460 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.480+0000 7f5bda231640 1 -- 172.21.15.105:0/3445528333 shutdown_connections 2024-03-29T11:09:01.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.480+0000 7f5bda231640 1 -- 172.21.15.105:0/3445528333 >> 172.21.15.105:0/3445528333 conn(0x7f5bd406b660 msgr2=0x7f5bd406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:01.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.482+0000 7f5bda231640 1 -- 172.21.15.105:0/3445528333 shutdown_connections 2024-03-29T11:09:01.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.482+0000 7f5bda231640 1 -- 172.21.15.105:0/3445528333 wait complete. 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.483+0000 7f5bda231640 10 monclient: build_initial_monmap 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.483+0000 7f5bda231640 10 monclient: monmap: 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.462 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:01.463 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:01.463 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:01.463 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 librados: starting msgr at 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 librados: starting objecter 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 Processor -- start 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 -- start start 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 librados: setting wanted keys 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 1 librados: calling monclient init 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.484+0000 7f5bda231640 10 monclient: init 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.485+0000 7f5bda231640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.485+0000 7f5bda231640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:01.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.485+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40ad850 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:01.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.488+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd40b0770 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:01.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.488+0000 7f5bd2ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f5bd40b0770 0x7f5bd41d3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:43835/0 (socket says 172.21.15.105:43835) 2024-03-29T11:09:01.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd413c1b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:01.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40705d0 con 0x7f5bd413c1b0 2024-03-29T11:09:01.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd41d3b60 con 0x7f5bd40ad850 2024-03-29T11:09:01.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd41d6300 con 0x7f5bd40b0770 2024-03-29T11:09:01.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:01.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bda231640 10 monclient(hunting): authenticate will time out at 815.765456s 2024-03-29T11:09:01.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.489+0000 7f5bd2ffd640 1 -- 172.21.15.105:0/1883176072 learned_addr learned my addr 172.21.15.105:0/1883176072 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:01.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 client.?.objecter ms_handle_connect 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328590843 0 0) 0x7f5bd41d6300 con 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0004eb0 con 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418501224 0 0) 0x7f5bc0004eb0 con 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd41d6300 con 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5bc8001950 con 0x7f5bd40b0770 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:01.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): dump: 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:01.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361762600 0 0) 0x7f5bd41d6300 con 0x7f5bd40b0770 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 >> v1:172.21.15.110:6789/0 conn(0x7f5bd40ad850 legacy=0x7f5bd406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.490+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 >> v1:172.21.15.105:6789/0 conn(0x7f5bd413c1b0 legacy=0x7f5bd41d6820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 1 monclient: found mon.c 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd406e780 con 0x7f5bd40b0770 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.491+0000 7f5bbbfff640 10 monclient: _finish_auth 0 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 10 monclient: _renew_subs 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:01.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd41d2c50 con 0x7f5bd40b0770 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 10 monclient: _renew_subs 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd41d7110 con 0x7f5bd40b0770 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bda231640 1 librados: init done 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc8001c70 con 0x7f5bd40b0770 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5bc8004c30 con 0x7f5bd40b0770 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bbbfff640 10 monclient: dump: 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:01.476 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.492+0000 7f5bd1ffb640 4 set_mon_vals no callback set 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.493+0000 7f5bda231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.493+0000 7f5bda231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.493+0000 7f5bda231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:01.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.493+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb4005080 con 0x7f5bd40b0770 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f5bc801ac00 con 0x7f5bd40b0770 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f5bc8084a30 con 0x7f5bd40b0770 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 10 client.4265.objecter ms_dispatch 0x7f5bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:01.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.495+0000 7f5bbbfff640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:09:01.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.497+0000 7f5bd37fe640 10 monclient: get_auth_request con 0x7f5bc006db60 auth_method 0 2024-03-29T11:09:01.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.498+0000 7f5bbbfff640 10 client.4265.objecter ms_handle_connect 0x7f5bc006db60 2024-03-29T11:09:01.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.502+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bc8050370 con 0x7f5bd40b0770 2024-03-29T11:09:01.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.502+0000 7f5bbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:01.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.502+0000 7f5bbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.831+0000 7f5bda231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.831+0000 7f5bda231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.831+0000 7f5bda231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.831+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bb4005330 con 0x7f5bd40b0770 2024-03-29T11:09:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.832+0000 7f5bbbfff640 1 -- 172.21.15.105:0/1883176072 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f5bc80522a0 con 0x7f5bd40b0770 2024-03-29T11:09:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.832+0000 7f5bbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.832+0000 7f5bbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:01.813 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:01.813 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:01.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.836+0000 7f5bda231640 10 librados: watch_flush enter 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 10 librados: watch_flush exit 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 >> v1:172.21.15.110:6816/1087970004 conn(0x7f5bc006db60 legacy=0x7f5bc0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 10 monclient: shutdown 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 >> v1:172.21.15.105:6790/0 conn(0x7f5bd40b0770 legacy=0x7f5bd41d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 shutdown_connections 2024-03-29T11:09:01.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.839+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 >> 172.21.15.105:0/1883176072 conn(0x7f5bd406b660 msgr2=0x7f5bd40aa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.840+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 shutdown_connections 2024-03-29T11:09:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.844+0000 7f5bda231640 1 -- 172.21.15.105:0/1883176072 wait complete. 2024-03-29T11:09:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:01.844+0000 7f5bda231640 1 librados: shutdown 2024-03-29T11:09:01.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:08.648 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.826+0000 7f505359e640 10 monclient: get_monmap_and_config 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.826+0000 7f505359e640 10 monclient: build_initial_monmap 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.826+0000 7f505359e640 10 monclient: monmap: 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:08.805 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:08.827130+0000 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:08.827130+0000 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:08.806 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.828+0000 7f505359e640 1 Processor -- start 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.828+0000 7f505359e640 1 -- start start 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.828+0000 7f505359e640 10 monclient: init 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient(hunting): picked mon.noname-c con 0x7f50540acfa0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient(hunting): picked mon.noname-a con 0x7f5054072f10 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient(hunting): picked mon.noname-b con 0x7f50540ad380 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50540aa9f0 con 0x7f5054072f10 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50540ac4c0 con 0x7f50540acfa0 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50540ac7a0 con 0x7f50540ad380 2024-03-29T11:09:08.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:08.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.829+0000 7f505359e640 10 monclient(hunting): authenticate will time out at 823.105430s 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f50540acfa0 0x7f5054141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:37181/0 (socket says 172.21.15.105:37181) 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505259c640 1 -- 172.21.15.105:0/2789519294 learned_addr learned my addr 172.21.15.105:0/2789519294 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90352094 0 0) 0x7f50540ac7a0 con 0x7f50540ad380 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c003520 con 0x7f50540ad380 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374137225 0 0) 0x7f503c003520 con 0x7f50540ad380 2024-03-29T11:09:08.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 >> v1:172.21.15.110:6789/0 conn(0x7f50540acfa0 legacy=0x7f5054141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:08.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.832+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 >> v1:172.21.15.105:6789/0 conn(0x7f5054072f10 legacy=0x7f5054141d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: _finish_hunting 0 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 1 monclient: found mon.noname-b 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50540ab5a0 con 0x7f50540ad380 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: _finish_auth 0 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: _check_auth_tickets 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505359e640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:09:08.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5044001690 con 0x7f50540ad380 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505159a640 10 monclient: dump: 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.836+0000 7f505359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5044001920 con 0x7f50540ad380 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 1 -- 172.21.15.105:0/2789519294 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5044001cb0 con 0x7f50540ad380 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505159a640 10 monclient: dump: 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:08.818 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.837+0000 7f505359e640 4 set_mon_vals no callback set 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.838+0000 7f505359e640 10 monclient: shutdown 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.838+0000 7f505359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.838+0000 7f505359e640 1 -- 172.21.15.105:0/2789519294 >> v1:172.21.15.105:6790/0 conn(0x7f50540ad380 legacy=0x7f50541497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.838+0000 7f505359e640 1 -- 172.21.15.105:0/2789519294 shutdown_connections 2024-03-29T11:09:08.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.838+0000 7f505359e640 1 -- 172.21.15.105:0/2789519294 >> 172.21.15.105:0/2789519294 conn(0x7f50540b3a30 msgr2=0x7f50540b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:08.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.840+0000 7f505359e640 1 -- 172.21.15.105:0/2789519294 shutdown_connections 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.840+0000 7f505359e640 1 -- 172.21.15.105:0/2789519294 wait complete. 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.841+0000 7f505359e640 10 monclient: build_initial_monmap 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.841+0000 7f505359e640 10 monclient: monmap: 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:08.821 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 librados: starting msgr at 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 librados: starting objecter 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 Processor -- start 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 -- start start 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 librados: setting wanted keys 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 1 librados: calling monclient init 2024-03-29T11:09:08.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.842+0000 7f505359e640 10 monclient: init 2024-03-29T11:09:08.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:08.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:08.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient(hunting): picked mon.noname-c con 0x7f5054072f10 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:08.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient(hunting): picked mon.noname-a con 0x7f50540acfa0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:08.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient(hunting): picked mon.noname-b con 0x7f50540ad380 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:08.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50541c80b0 con 0x7f50540acfa0 2024-03-29T11:09:08.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50541c7150 con 0x7f50540ad380 2024-03-29T11:09:08.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50541c7c80 con 0x7f5054072f10 2024-03-29T11:09:08.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:08.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.843+0000 7f505359e640 10 monclient(hunting): authenticate will time out at 823.119396s 2024-03-29T11:09:08.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.848+0000 7f5051d9b640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f50540acfa0 0x7f50541c9bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:45457/0 (socket says 172.21.15.105:45457) 2024-03-29T11:09:08.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.848+0000 7f5051d9b640 1 -- 172.21.15.105:0/1289506117 learned_addr learned my addr 172.21.15.105:0/1289506117 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:08.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.848+0000 7f5042ffd640 10 client.?.objecter ms_handle_connect 0x7f50540acfa0 2024-03-29T11:09:08.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.848+0000 7f5042ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469140168 0 0) 0x7f50541c80b0 con 0x7f50540acfa0 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5034004eb0 con 0x7f50540acfa0 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 client.?.objecter ms_handle_connect 0x7f50540ad380 2024-03-29T11:09:08.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.850+0000 7f5042ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645388283 0 0) 0x7f5034004eb0 con 0x7f50540acfa0 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50541c80b0 con 0x7f50540acfa0 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5048004050 con 0x7f50540acfa0 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.831 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.852+0000 7f5042ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:08.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725931906 0 0) 0x7f50541c7150 con 0x7f50540ad380 2024-03-29T11:09:08.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5034004eb0 con 0x7f50540ad380 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4031514325 0 0) 0x7f50541c80b0 con 0x7f50540acfa0 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 >> v1:172.21.15.105:6790/0 conn(0x7f5054072f10 legacy=0x7f5054071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 >> v1:172.21.15.110:6789/0 conn(0x7f50540ad380 legacy=0x7f5054136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 monclient: found mon.a 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50541c78c0 con 0x7f50540acfa0 2024-03-29T11:09:08.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f5042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 10 monclient: _renew_subs 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50541c8540 con 0x7f50540acfa0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 10 monclient: _renew_subs 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50541ca4c0 con 0x7f50540acfa0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.853+0000 7f505359e640 1 librados: init done 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5048000ed0 con 0x7f50540acfa0 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:08.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f50480012a0 con 0x7f50540acfa0 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f5042ffd640 10 monclient: dump: 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:08.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f505359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:08.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f505359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:08.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f505359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:08.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.854+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50541c7e10 con 0x7f50540acfa0 2024-03-29T11:09:08.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.856+0000 7f5050d99640 4 set_mon_vals no callback set 2024-03-29T11:09:08.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.860+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f5048001400 con 0x7f50540acfa0 2024-03-29T11:09:08.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f505259c640 10 monclient: get_auth_request con 0x7f503406dd70 auth_method 0 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f5048084a70 con 0x7f50540acfa0 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 10 client.4251.objecter ms_dispatch 0x7f5054000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:09:08.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.861+0000 7f5042ffd640 10 client.4251.objecter ms_handle_connect 0x7f503406dd70 2024-03-29T11:09:08.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.866+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5048050350 con 0x7f50540acfa0 2024-03-29T11:09:08.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.866+0000 7f5042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:08.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:08.866+0000 7f5042ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.205+0000 7f505359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.205+0000 7f505359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.205+0000 7f505359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.205+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f505404f730 con 0x7f50540acfa0 2024-03-29T11:09:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.208+0000 7f5042ffd640 1 -- 172.21.15.105:0/1289506117 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f5048052280 con 0x7f50540acfa0 2024-03-29T11:09:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.208+0000 7f5042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.208+0000 7f5042ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:09.187 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:09.187 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:09.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.211+0000 7f505359e640 10 librados: watch_flush enter 2024-03-29T11:09:09.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.211+0000 7f505359e640 10 librados: watch_flush exit 2024-03-29T11:09:09.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.211+0000 7f505359e640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:09:09.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.211+0000 7f505359e640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 >> v1:172.21.15.110:6816/1087970004 conn(0x7f503406dd70 legacy=0x7f5034070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 10 monclient: shutdown 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 >> v1:172.21.15.105:6789/0 conn(0x7f50540acfa0 legacy=0x7f50541c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 shutdown_connections 2024-03-29T11:09:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.213+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 >> 172.21.15.105:0/1289506117 conn(0x7f50540b3a30 msgr2=0x7f50540b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.216+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 shutdown_connections 2024-03-29T11:09:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.216+0000 7f505359e640 1 -- 172.21.15.105:0/1289506117 wait complete. 2024-03-29T11:09:09.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:09.218+0000 7f505359e640 1 librados: shutdown 2024-03-29T11:09:09.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:16.023 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:16.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.191+0000 7f440ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:09:16.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.192+0000 7f440ffff640 10 monclient: build_initial_monmap 2024-03-29T11:09:16.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.192+0000 7f440ffff640 10 monclient: monmap: 2024-03-29T11:09:16.171 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:16.193083+0000 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:16.193083+0000 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.193+0000 7f440ffff640 1 Processor -- start 2024-03-29T11:09:16.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.193+0000 7f440ffff640 1 -- start start 2024-03-29T11:09:16.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.193+0000 7f440ffff640 10 monclient: init 2024-03-29T11:09:16.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.194+0000 7f440ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:16.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.194+0000 7f440ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:16.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.194+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4408034850 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:16.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.197+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f44080c9550 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:16.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f440802aac0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:16.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440802c440 con 0x7f4408034850 2024-03-29T11:09:16.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440802aed0 con 0x7f44080c9550 2024-03-29T11:09:16.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440802b190 con 0x7f440802aac0 2024-03-29T11:09:16.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:16.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440ffff640 10 monclient(hunting): authenticate will time out at 830.474034s 2024-03-29T11:09:16.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440e7fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f44080c9550 0x7f44080cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:54873/0 (socket says 172.21.15.105:54873) 2024-03-29T11:09:16.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.198+0000 7f440e7fc640 1 -- 172.21.15.105:0/3596648819 learned_addr learned my addr 172.21.15.105:0/3596648819 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574793124 0 0) 0x7f440802aed0 con 0x7f44080c9550 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f8003520 con 0x7f44080c9550 2024-03-29T11:09:16.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551895689 0 0) 0x7f440802b190 con 0x7f440802aac0 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440802aed0 con 0x7f440802aac0 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.201+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591184895 0 0) 0x7f43f8003520 con 0x7f44080c9550 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 >> v1:172.21.15.105:6790/0 conn(0x7f440802aac0 legacy=0x7f44080cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 >> v1:172.21.15.105:6789/0 conn(0x7f4408034850 legacy=0x7f44080c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 monclient: found mon.noname-c 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408033060 con 0x7f44080c9550 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:16.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f441004f730 con 0x7f44080c9550 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: dump: 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:16.182 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:16.183 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440ffff640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:09:16.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:16.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44100008d0 con 0x7f44080c9550 2024-03-29T11:09:16.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:16.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 1 -- 172.21.15.105:0/3596648819 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f44100758b0 con 0x7f44080c9550 2024-03-29T11:09:16.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440dffb640 10 monclient: dump: 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:16.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.202+0000 7f440ffff640 4 set_mon_vals no callback set 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.203+0000 7f440ffff640 10 monclient: shutdown 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.203+0000 7f440ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.203+0000 7f440ffff640 1 -- 172.21.15.105:0/3596648819 >> v1:172.21.15.110:6789/0 conn(0x7f44080c9550 legacy=0x7f44080cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.204+0000 7f440ffff640 1 -- 172.21.15.105:0/3596648819 shutdown_connections 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.204+0000 7f440ffff640 1 -- 172.21.15.105:0/3596648819 >> 172.21.15.105:0/3596648819 conn(0x7f44080058f0 msgr2=0x7f4408005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:16.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.205+0000 7f440ffff640 1 -- 172.21.15.105:0/3596648819 shutdown_connections 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.206+0000 7f440ffff640 1 -- 172.21.15.105:0/3596648819 wait complete. 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.207+0000 7f440ffff640 10 monclient: build_initial_monmap 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.207+0000 7f440ffff640 10 monclient: monmap: 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:16.187 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 librados: starting msgr at 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 librados: starting objecter 2024-03-29T11:09:16.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 Processor -- start 2024-03-29T11:09:16.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 -- start start 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 librados: setting wanted keys 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 1 librados: calling monclient init 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.208+0000 7f440ffff640 10 monclient: init 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.209+0000 7f440ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.209+0000 7f440ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.209+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f440802aac0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4408034850 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f44080c9550 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44081655e0 con 0x7f4408034850 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44080be5f0 con 0x7f44080c9550 2024-03-29T11:09:16.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440802d890 con 0x7f440802aac0 2024-03-29T11:09:16.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:16.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.212+0000 7f440ffff640 10 monclient(hunting): authenticate will time out at 830.488077s 2024-03-29T11:09:16.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.214+0000 7f440f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f44080c9550 0x7f4408164ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:54969/0 (socket says 172.21.15.105:54969) 2024-03-29T11:09:16.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.214+0000 7f440f7fe640 1 -- 172.21.15.105:0/2939477588 learned_addr learned my addr 172.21.15.105:0/2939477588 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:16.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.216+0000 7f43f77fe640 10 client.?.objecter ms_handle_connect 0x7f4408034850 2024-03-29T11:09:16.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.216+0000 7f43f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.216+0000 7f43f77fe640 10 client.?.objecter ms_handle_connect 0x7f440802aac0 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.216+0000 7f43f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072268414 0 0) 0x7f440802d890 con 0x7f440802aac0 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:09:16.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.217+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43fc004eb0 con 0x7f440802aac0 2024-03-29T11:09:16.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.218+0000 7f43f77fe640 10 client.?.objecter ms_handle_connect 0x7f44080c9550 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.218+0000 7f43f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998082720 0 0) 0x7f43fc004eb0 con 0x7f440802aac0 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f440802d890 con 0x7f440802aac0 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4404001870 con 0x7f440802aac0 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): dump: 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:16.198 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179695554 0 0) 0x7f44081655e0 con 0x7f4408034850 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43fc004eb0 con 0x7f4408034850 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501972680 0 0) 0x7f440802d890 con 0x7f440802aac0 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 >> v1:172.21.15.110:6789/0 conn(0x7f44080c9550 legacy=0x7f4408164ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 >> v1:172.21.15.105:6789/0 conn(0x7f4408034850 legacy=0x7f44081647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 monclient: found mon.c 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440802db70 con 0x7f440802aac0 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:16.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f43f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:16.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 10 monclient: _renew_subs 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440802de50 con 0x7f440802aac0 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 10 monclient: _renew_subs 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f440802caf0 con 0x7f440802aac0 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.219+0000 7f440ffff640 1 librados: init done 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.220+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44040035f0 con 0x7f440802aac0 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.220+0000 7f43f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.220+0000 7f440d7fa640 4 set_mon_vals no callback set 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f440ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:16.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f44040019d0 con 0x7f440802aac0 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f43f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f43f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f43f77fe640 10 monclient: dump: 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:16.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f440ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f440ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.221+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f0005080 con 0x7f440802aac0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.222+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f4404003a00 con 0x7f440802aac0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f4404083840 con 0x7f440802aac0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f43f77fe640 10 client.4274.objecter ms_dispatch 0x7f4410000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f43f77fe640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f43f77fe640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f43f77fe640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.223+0000 7f440e7fc640 10 monclient: get_auth_request con 0x7f43fc06dcf0 auth_method 0 2024-03-29T11:09:16.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.225+0000 7f43f77fe640 10 client.4274.objecter ms_handle_connect 0x7f43fc06dcf0 2024-03-29T11:09:16.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.230+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f440404ef50 con 0x7f440802aac0 2024-03-29T11:09:16.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.230+0000 7f43f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:16.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.230+0000 7f43f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:16.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.559+0000 7f440ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:16.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.559+0000 7f440ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:16.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.559+0000 7f440ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:16.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.559+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43f0005330 con 0x7f440802aac0 2024-03-29T11:09:16.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.560+0000 7f43f77fe640 1 -- 172.21.15.105:0/2939477588 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f4404051050 con 0x7f440802aac0 2024-03-29T11:09:16.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.560+0000 7f43f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:16.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.560+0000 7f43f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:16.541 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:16.541 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:16.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 10 librados: watch_flush enter 2024-03-29T11:09:16.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 10 librados: watch_flush exit 2024-03-29T11:09:16.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:09:16.543 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:09:16.543 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 >> v1:172.21.15.110:6816/1087970004 conn(0x7f43fc06dcf0 legacy=0x7f43fc0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.543 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 10 monclient: shutdown 2024-03-29T11:09:16.543 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.564+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 >> v1:172.21.15.105:6790/0 conn(0x7f440802aac0 legacy=0x7f440802e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.565+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 shutdown_connections 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.565+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 >> 172.21.15.105:0/2939477588 conn(0x7f44080058f0 msgr2=0x7f44080043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.565+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 shutdown_connections 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.565+0000 7f440ffff640 1 -- 172.21.15.105:0/2939477588 wait complete. 2024-03-29T11:09:16.544 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:16.565+0000 7f440ffff640 1 librados: shutdown 2024-03-29T11:09:16.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:23.370 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.539+0000 7f76afd72640 10 monclient: get_monmap_and_config 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.539+0000 7f76afd72640 10 monclient: build_initial_monmap 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.539+0000 7f76afd72640 10 monclient: monmap: 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:23.518 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:23.540782+0000 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:23.540782+0000 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:23.519 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.541+0000 7f76afd72640 1 Processor -- start 2024-03-29T11:09:23.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.541+0000 7f76afd72640 1 -- start start 2024-03-29T11:09:23.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.541+0000 7f76afd72640 10 monclient: init 2024-03-29T11:09:23.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:23.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-c con 0x7f76a80b4330 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-b con 0x7f76a81470b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-a con 0x7f76a81439b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a806f2a0 con 0x7f76a81439b0 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8146b90 con 0x7f76a80b4330 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80b2ca0 con 0x7f76a81470b0 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:23.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.542+0000 7f76afd72640 10 monclient(hunting): authenticate will time out at 837.818789s 2024-03-29T11:09:23.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.546+0000 7f76adb0e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f76a80b4330 0x7f76a80b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:40417/0 (socket says 172.21.15.105:40417) 2024-03-29T11:09:23.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.546+0000 7f76adb0e640 1 -- 172.21.15.105:0/1843050400 learned_addr learned my addr 172.21.15.105:0/1843050400 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:23.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384149420 0 0) 0x7f76a80b2ca0 con 0x7f76a81470b0 2024-03-29T11:09:23.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _negotiate 2024-03-29T11:09:23.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:23.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:23.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:09:23.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7694003520 con 0x7f76a81470b0 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120297563 0 0) 0x7f76a806f2a0 con 0x7f76a81439b0 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _negotiate 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a80b2ca0 con 0x7f76a81439b0 2024-03-29T11:09:23.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567966275 0 0) 0x7f7694003520 con 0x7f76a81470b0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 >> v1:172.21.15.110:6789/0 conn(0x7f76a80b4330 legacy=0x7f76a80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 >> v1:172.21.15.105:6789/0 conn(0x7f76a81439b0 legacy=0x7f76a8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: _finish_hunting 0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 monclient: found mon.noname-b 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a806e0c0 con 0x7f76a81470b0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: _finish_auth 0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: _check_auth_tickets 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7698001690 con 0x7f76a81470b0 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:23.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.548+0000 7f76acb0c640 10 monclient: dump: 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76afd72640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76afd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76980017f0 con 0x7f76a81470b0 2024-03-29T11:09:23.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 1 -- 172.21.15.105:0/1843050400 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7698001c00 con 0x7f76a81470b0 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.551+0000 7f76acb0c640 10 monclient: dump: 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:23.534 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.553+0000 7f76afd72640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.553+0000 7f76afd72640 4 set_mon_vals no callback set 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.554+0000 7f76afd72640 10 monclient: shutdown 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.554+0000 7f76afd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.554+0000 7f76afd72640 1 -- 172.21.15.105:0/1843050400 >> v1:172.21.15.105:6790/0 conn(0x7f76a81470b0 legacy=0x7f76a80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.556+0000 7f76afd72640 1 -- 172.21.15.105:0/1843050400 shutdown_connections 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.556+0000 7f76afd72640 1 -- 172.21.15.105:0/1843050400 >> 172.21.15.105:0/1843050400 conn(0x7f76a8066a00 msgr2=0x7f76a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:23.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.556+0000 7f76afd72640 1 -- 172.21.15.105:0/1843050400 shutdown_connections 2024-03-29T11:09:23.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.560+0000 7f76afd72640 1 -- 172.21.15.105:0/1843050400 wait complete. 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.560+0000 7f76afd72640 10 monclient: build_initial_monmap 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.560+0000 7f76afd72640 10 monclient: monmap: 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:23.539 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 librados: starting msgr at 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 librados: starting objecter 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 Processor -- start 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 -- start start 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 librados: setting wanted keys 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 1 librados: calling monclient init 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.561+0000 7f76afd72640 10 monclient: init 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.562+0000 7f76afd72640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:23.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.562+0000 7f76afd72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-a con 0x7f76a80b4330 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-b con 0x7f76a81439b0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 10 monclient(hunting): picked mon.noname-c con 0x7f76a81470b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8121660 con 0x7f76a80b4330 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8121940 con 0x7f76a81439b0 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8120490 con 0x7f76a81470b0 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:23.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.566+0000 7f76afd72640 10 monclient(hunting): authenticate will time out at 837.842771s 2024-03-29T11:09:23.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f76adb0e640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f76a80b4330 0x7f76a81407e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:56385/0 (socket says 172.21.15.105:56385) 2024-03-29T11:09:23.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f76adb0e640 1 -- 172.21.15.105:0/1578852225 learned_addr learned my addr 172.21.15.105:0/1578852225 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:23.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 client.?.objecter ms_handle_connect 0x7f76a80b4330 2024-03-29T11:09:23.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280384313 0 0) 0x7f76a8121660 con 0x7f76a80b4330 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c004eb0 con 0x7f76a80b4330 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.569+0000 7f769e7fc640 10 client.?.objecter ms_handle_connect 0x7f76a81439b0 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 client.?.objecter ms_handle_connect 0x7f76a81470b0 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335303237 0 0) 0x7f768c004eb0 con 0x7f76a80b4330 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a8121660 con 0x7f76a80b4330 2024-03-29T11:09:23.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f76a0001b90 con 0x7f76a80b4330 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): dump: 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.550 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:23.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506304748 0 0) 0x7f76a8121940 con 0x7f76a81439b0 2024-03-29T11:09:23.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c004eb0 con 0x7f76a81439b0 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380509237 0 0) 0x7f76a8120490 con 0x7f76a81470b0 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a8121940 con 0x7f76a81470b0 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 768696453 0 0) 0x7f76a8121660 con 0x7f76a80b4330 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 >> v1:172.21.15.105:6790/0 conn(0x7f76a81470b0 legacy=0x7f76a81df580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 >> v1:172.21.15.110:6789/0 conn(0x7f76a81439b0 legacy=0x7f76a81dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 monclient: found mon.a 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8120120 con 0x7f76a80b4330 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.570+0000 7f769e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.571+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a0001d70 con 0x7f76a80b4330 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.571+0000 7f769e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.571+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f76a0004dc0 con 0x7f76a80b4330 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.571+0000 7f769f7fe640 4 set_mon_vals no callback set 2024-03-29T11:09:23.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.572+0000 7f769e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.572+0000 7f769e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.572+0000 7f769e7fc640 10 monclient: dump: 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:23.554 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 10 monclient: _renew_subs 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a8120900 con 0x7f76a80b4330 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 10 monclient: _renew_subs 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a81410d0 con 0x7f76a80b4330 2024-03-29T11:09:23.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.573+0000 7f76afd72640 1 librados: init done 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.576+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f76a0004f20 con 0x7f76a80b4330 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.576+0000 7f76afd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f76afd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f76afd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a8120620 con 0x7f76a80b4330 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f76a0084f20 con 0x7f76a80b4330 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f769e7fc640 10 client.4260.objecter ms_dispatch 0x7f76a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f769e7fc640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f769e7fc640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f76ad30d640 10 monclient: get_auth_request con 0x7f768c06de80 auth_method 0 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.577+0000 7f769e7fc640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:09:23.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.578+0000 7f769e7fc640 10 client.4260.objecter ms_handle_connect 0x7f768c06de80 2024-03-29T11:09:23.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.586+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76a0050800 con 0x7f76a80b4330 2024-03-29T11:09:23.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.586+0000 7f769e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:23.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.586+0000 7f769e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:23.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.914+0000 7f76afd72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:23.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.914+0000 7f76afd72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:23.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.914+0000 7f76afd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:23.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.914+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76a8120a90 con 0x7f76a80b4330 2024-03-29T11:09:23.894 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.916+0000 7f769e7fc640 1 -- 172.21.15.105:0/1578852225 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f76a0052730 con 0x7f76a80b4330 2024-03-29T11:09:23.894 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.916+0000 7f769e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:23.894 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.916+0000 7f769e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:23.896 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:23.896 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:23.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.920+0000 7f76afd72640 10 librados: watch_flush enter 2024-03-29T11:09:23.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 10 librados: watch_flush exit 2024-03-29T11:09:23.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:09:23.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:09:23.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 >> v1:172.21.15.110:6816/1087970004 conn(0x7f768c06de80 legacy=0x7f768c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 10 monclient: shutdown 2024-03-29T11:09:23.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:23.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 >> v1:172.21.15.105:6789/0 conn(0x7f76a80b4330 legacy=0x7f76a81407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:23.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 shutdown_connections 2024-03-29T11:09:23.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.924+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 >> 172.21.15.105:0/1578852225 conn(0x7f76a8066a00 msgr2=0x7f76a81d14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:23.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.925+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 shutdown_connections 2024-03-29T11:09:23.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.925+0000 7f76afd72640 1 -- 172.21.15.105:0/1578852225 wait complete. 2024-03-29T11:09:23.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:23.925+0000 7f76afd72640 1 librados: shutdown 2024-03-29T11:09:23.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:30.726 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.895+0000 7fb868cff640 10 monclient: get_monmap_and_config 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.895+0000 7fb868cff640 10 monclient: build_initial_monmap 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.896+0000 7fb868cff640 10 monclient: monmap: 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:30.897020+0000 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:30.897020+0000 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.877 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.897+0000 7fb868cff640 1 Processor -- start 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.897+0000 7fb868cff640 1 -- start start 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.897+0000 7fb868cff640 10 monclient: init 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-a con 0x7fb8640b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-b con 0x7fb86414a700 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-c con 0x7fb8640ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640aba70 con 0x7fb8640b0770 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640aecb0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640af150 con 0x7fb86414a700 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:30.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.898+0000 7fb868cff640 10 monclient(hunting): authenticate will time out at 845.174270s 2024-03-29T11:09:30.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.902+0000 7fb861d9b640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fb86414a700 0x7fb8640abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60169/0 (socket says 172.21.15.105:60169) 2024-03-29T11:09:30.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.902+0000 7fb861d9b640 1 -- 172.21.15.105:0/2052080280 learned_addr learned my addr 172.21.15.105:0/2052080280 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:30.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423230536 0 0) 0x7fb8640af150 con 0x7fb86414a700 2024-03-29T11:09:30.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:30.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:30.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:30.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:09:30.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003520 con 0x7fb86414a700 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743893296 0 0) 0x7fb84c003520 con 0x7fb86414a700 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 >> v1:172.21.15.110:6789/0 conn(0x7fb8640ac5a0 legacy=0x7fb864151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 >> v1:172.21.15.105:6789/0 conn(0x7fb8640b0770 legacy=0x7fb864149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient: _finish_hunting 0 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 monclient: found mon.noname-b 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8640ac9d0 con 0x7fb86414a700 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient: _finish_auth 0 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 10 monclient: _check_auth_tickets 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.905+0000 7fb86159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb868cff640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:09:30.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb868cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb85c001a10 con 0x7fb86414a700 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: dump: 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb868cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:30.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb85c001b70 con 0x7fb86414a700 2024-03-29T11:09:30.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb868cff640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb868cff640 4 set_mon_vals no callback set 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 1 -- 172.21.15.105:0/2052080280 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb85c001fc0 con 0x7fb86414a700 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.906+0000 7fb86159a640 10 monclient: dump: 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:30.889 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 10 monclient: shutdown 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 1 -- 172.21.15.105:0/2052080280 >> v1:172.21.15.105:6790/0 conn(0x7fb86414a700 legacy=0x7fb8640abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 1 -- 172.21.15.105:0/2052080280 shutdown_connections 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 1 -- 172.21.15.105:0/2052080280 >> 172.21.15.105:0/2052080280 conn(0x7fb86406b660 msgr2=0x7fb86406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.907+0000 7fb868cff640 1 -- 172.21.15.105:0/2052080280 shutdown_connections 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.908+0000 7fb868cff640 1 -- 172.21.15.105:0/2052080280 wait complete. 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.909+0000 7fb868cff640 10 monclient: build_initial_monmap 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.909+0000 7fb868cff640 10 monclient: monmap: 2024-03-29T11:09:30.890 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.909+0000 7fb868cff640 1 librados: starting msgr at 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.909+0000 7fb868cff640 1 librados: starting objecter 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.909+0000 7fb868cff640 1 Processor -- start 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 -- start start 2024-03-29T11:09:30.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 librados: setting wanted keys 2024-03-29T11:09:30.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 librados: calling monclient init 2024-03-29T11:09:30.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient: init 2024-03-29T11:09:30.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:30.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-c con 0x7fb8640ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-a con 0x7fb8640b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient(hunting): picked mon.noname-b con 0x7fb86414a700 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8641decc0 con 0x7fb8640b0770 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8641285e0 con 0x7fb86414a700 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8641239d0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.910+0000 7fb868cff640 10 monclient(hunting): authenticate will time out at 845.186937s 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.915+0000 7fb862d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb86414a700 0x7fb8641df150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:37589/0 (socket says 172.21.15.105:37589) 2024-03-29T11:09:30.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.915+0000 7fb862d9d640 1 -- 172.21.15.105:0/1301035945 learned_addr learned my addr 172.21.15.105:0/1301035945 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 client.?.objecter ms_handle_connect 0x7fb86414a700 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502206555 0 0) 0x7fb8641285e0 con 0x7fb86414a700 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844004eb0 con 0x7fb86414a700 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.917+0000 7fb852ffd640 10 client.?.objecter ms_handle_connect 0x7fb8640ac5a0 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753544650 0 0) 0x7fb8641239d0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:30.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8641285e0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 213687402 0 0) 0x7fb844004eb0 con 0x7fb86414a700 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8641239d0 con 0x7fb86414a700 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb8580018e0 con 0x7fb86414a700 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.898 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553395324 0 0) 0x7fb8641285e0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb844004eb0 con 0x7fb8640ac5a0 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1104751660 0 0) 0x7fb8641239d0 con 0x7fb86414a700 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 >> v1:172.21.15.105:6790/0 conn(0x7fb8640ac5a0 legacy=0x7fb864126f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 >> v1:172.21.15.105:6789/0 conn(0x7fb8640b0770 legacy=0x7fb8641261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 monclient: found mon.b 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:30.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.918+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864123b60 con 0x7fb86414a700 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 10 monclient: _renew_subs 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:30.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641df860 con 0x7fb86414a700 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 10 monclient: _renew_subs 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8641453f0 con 0x7fb86414a700 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb868cff640 1 librados: init done 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb858001ac0 con 0x7fb86414a700 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb8580038c0 con 0x7fb86414a700 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb852ffd640 10 monclient: dump: 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:30.902 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.919+0000 7fb860d99640 4 set_mon_vals no callback set 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.921+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb858003a20 con 0x7fb86414a700 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fb858084a00 con 0x7fb86414a700 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb852ffd640 10 client.4273.objecter ms_dispatch 0x7fb864000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb852ffd640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb852ffd640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb852ffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T11:09:30.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb868cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:30.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb868cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:30.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb868cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:30.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb848005080 con 0x7fb86414a700 2024-03-29T11:09:30.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.922+0000 7fb86259c640 10 monclient: get_auth_request con 0x7fb84406de40 auth_method 0 2024-03-29T11:09:30.904 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.923+0000 7fb852ffd640 10 client.4273.objecter ms_handle_connect 0x7fb84406de40 2024-03-29T11:09:30.909 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.931+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb858050110 con 0x7fb86414a700 2024-03-29T11:09:30.909 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.931+0000 7fb852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:30.912 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:30.933+0000 7fb852ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:31.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.263+0000 7fb868cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:31.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.263+0000 7fb868cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:31.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.263+0000 7fb868cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:31.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.263+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb848005330 con 0x7fb86414a700 2024-03-29T11:09:31.245 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:31.245 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:31.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.266+0000 7fb852ffd640 1 -- 172.21.15.105:0/1301035945 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fb858052210 con 0x7fb86414a700 2024-03-29T11:09:31.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.266+0000 7fb852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:31.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.266+0000 7fb852ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.270+0000 7fb868cff640 10 librados: watch_flush enter 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.270+0000 7fb868cff640 10 librados: watch_flush exit 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.270+0000 7fb868cff640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.270+0000 7fb868cff640 10 client.4273.objecter successfully canceled tick 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.271+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb84406de40 legacy=0x7fb844070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:31.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.271+0000 7fb868cff640 10 monclient: shutdown 2024-03-29T11:09:31.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.271+0000 7fb868cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:31.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.271+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 >> v1:172.21.15.110:6789/0 conn(0x7fb86414a700 legacy=0x7fb8641df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:31.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.271+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 shutdown_connections 2024-03-29T11:09:31.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.272+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 >> 172.21.15.105:0/1301035945 conn(0x7fb86406b660 msgr2=0x7fb8640b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:31.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.272+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 shutdown_connections 2024-03-29T11:09:31.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.272+0000 7fb868cff640 1 -- 172.21.15.105:0/1301035945 wait complete. 2024-03-29T11:09:31.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:31.272+0000 7fb868cff640 1 librados: shutdown 2024-03-29T11:09:31.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:38.077 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:38.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.262+0000 7ff056f6b640 10 monclient: get_monmap_and_config 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.262+0000 7ff056f6b640 10 monclient: build_initial_monmap 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.262+0000 7ff056f6b640 10 monclient: monmap: 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:38.263437+0000 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:38.263437+0000 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:38.241 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:38.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.268+0000 7ff056f6b640 1 Processor -- start 2024-03-29T11:09:38.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.268+0000 7ff056f6b640 1 -- start start 2024-03-29T11:09:38.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.268+0000 7ff056f6b640 10 monclient: init 2024-03-29T11:09:38.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.270+0000 7ff056f6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:38.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.270+0000 7ff056f6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.272+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff0501457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff05013f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff0500a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500aaef0 con 0x7ff0501457f0 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050145bd0 con 0x7ff0500a9d70 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050145eb0 con 0x7ff05013f650 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:38.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.273+0000 7ff056f6b640 10 monclient(hunting): authenticate will time out at 852.549375s 2024-03-29T11:09:38.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.275+0000 7ff054d07640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff0501457f0 0x7ff05013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:51947/0 (socket says 172.21.15.105:51947) 2024-03-29T11:09:38.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.275+0000 7ff054d07640 1 -- 172.21.15.105:0/1160522431 learned_addr learned my addr 172.21.15.105:0/1160522431 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:38.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880399304 0 0) 0x7ff0500aaef0 con 0x7ff0501457f0 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff040003520 con 0x7ff0501457f0 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319891808 0 0) 0x7ff040003520 con 0x7ff0501457f0 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 >> v1:172.21.15.105:6790/0 conn(0x7ff05013f650 legacy=0x7ff0500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:38.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.276+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 >> v1:172.21.15.110:6789/0 conn(0x7ff0500a9d70 legacy=0x7ff05014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050146340 con 0x7ff0501457f0 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff056f6b640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff056f6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff044001690 con 0x7ff0501457f0 2024-03-29T11:09:38.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff04f7fe640 10 monclient: dump: 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:38.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.277+0000 7ff056f6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.278+0000 7ff04f7fe640 1 -- 172.21.15.105:0/1160522431 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff044001920 con 0x7ff0501457f0 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.278+0000 7ff04f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.278+0000 7ff056f6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.278+0000 7ff056f6b640 4 set_mon_vals no callback set 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.279+0000 7ff056f6b640 10 monclient: shutdown 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.279+0000 7ff056f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:38.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.279+0000 7ff056f6b640 1 -- 172.21.15.105:0/1160522431 >> v1:172.21.15.105:6789/0 conn(0x7ff0501457f0 legacy=0x7ff05013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:38.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.281+0000 7ff056f6b640 1 -- 172.21.15.105:0/1160522431 shutdown_connections 2024-03-29T11:09:38.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.281+0000 7ff056f6b640 1 -- 172.21.15.105:0/1160522431 >> 172.21.15.105:0/1160522431 conn(0x7ff050066a00 msgr2=0x7ff050066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:38.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.283+0000 7ff056f6b640 1 -- 172.21.15.105:0/1160522431 shutdown_connections 2024-03-29T11:09:38.262 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.284+0000 7ff056f6b640 1 -- 172.21.15.105:0/1160522431 wait complete. 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.285+0000 7ff056f6b640 10 monclient: build_initial_monmap 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.286+0000 7ff056f6b640 10 monclient: monmap: 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:38.264 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:38.265 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.286+0000 7ff056f6b640 1 librados: starting msgr at 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.286+0000 7ff056f6b640 1 librados: starting objecter 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.286+0000 7ff056f6b640 1 Processor -- start 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.287+0000 7ff056f6b640 1 -- start start 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.287+0000 7ff056f6b640 1 librados: setting wanted keys 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.287+0000 7ff056f6b640 1 librados: calling monclient init 2024-03-29T11:09:38.266 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.287+0000 7ff056f6b640 10 monclient: init 2024-03-29T11:09:38.267 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:38.267 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:38.267 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff0500a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:38.267 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff05013f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff0501457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500b5c90 con 0x7ff0500a9d70 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500ad300 con 0x7ff0501457f0 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500ad550 con 0x7ff05013f650 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:38.268 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.288+0000 7ff056f6b640 10 monclient(hunting): authenticate will time out at 852.564692s 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.293+0000 7ff054d07640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff0500a9d70 0x7ff0501da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:52169/0 (socket says 172.21.15.105:52169) 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.294+0000 7ff054d07640 1 -- 172.21.15.105:0/2412702720 learned_addr learned my addr 172.21.15.105:0/2412702720 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.296+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff0500a9d70 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.296+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098264675 0 0) 0x7ff0500b5c90 con 0x7ff0500a9d70 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:09:38.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.297+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff034004eb0 con 0x7ff0500a9d70 2024-03-29T11:09:38.281 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff05013f650 2024-03-29T11:09:38.281 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:38.281 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff0501457f0 2024-03-29T11:09:38.281 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:38.281 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810365815 0 0) 0x7ff0500ad300 con 0x7ff0501457f0 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.300+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0500b5c90 con 0x7ff0501457f0 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418366502 0 0) 0x7ff0500ad550 con 0x7ff05013f650 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0500ad300 con 0x7ff05013f650 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181809482 0 0) 0x7ff034004eb0 con 0x7ff0500a9d70 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0500ad550 con 0x7ff0500a9d70 2024-03-29T11:09:38.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff0440049c0 con 0x7ff0500a9d70 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): dump: 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.301+0000 7ff04d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:38.283 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 485386531 0 0) 0x7ff0500b5c90 con 0x7ff0501457f0 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff034004eb0 con 0x7ff0501457f0 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 648435496 0 0) 0x7ff0500ad550 con 0x7ff0500a9d70 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 >> v1:172.21.15.105:6790/0 conn(0x7ff05013f650 legacy=0x7ff0500b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 >> v1:172.21.15.110:6789/0 conn(0x7ff0501457f0 legacy=0x7ff0501e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.302+0000 7ff04d7fa640 1 monclient: found mon.a 2024-03-29T11:09:38.284 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.303+0000 7ff04d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0500ac350 con 0x7ff0500a9d70 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:09:38.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 10 monclient: _renew_subs 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0500b86e0 con 0x7ff0500a9d70 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 10 monclient: _renew_subs 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0500abb70 con 0x7ff0500a9d70 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff056f6b640 1 librados: init done 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff044004f00 con 0x7ff0500a9d70 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.305+0000 7ff04effd640 4 set_mon_vals no callback set 2024-03-29T11:09:38.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff044005290 con 0x7ff0500a9d70 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff04d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff04d7fa640 10 monclient: dump: 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:38.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff056f6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:38.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff056f6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:38.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff056f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.306+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff030005080 con 0x7ff0500a9d70 2024-03-29T11:09:38.289 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.309+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff044001b10 con 0x7ff0500a9d70 2024-03-29T11:09:38.289 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.310+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff044084be0 con 0x7ff0500a9d70 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.311+0000 7ff04d7fa640 10 client.4269.objecter ms_dispatch 0x7ff050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.311+0000 7ff04d7fa640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.311+0000 7ff04d7fa640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.311+0000 7ff04d7fa640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.311+0000 7ff04ffff640 10 monclient: get_auth_request con 0x7ff03406df50 auth_method 0 2024-03-29T11:09:38.290 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.312+0000 7ff04d7fa640 10 client.4269.objecter ms_handle_connect 0x7ff03406df50 2024-03-29T11:09:38.300 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.322+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0440504c0 con 0x7ff0500a9d70 2024-03-29T11:09:38.300 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.322+0000 7ff04d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:38.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.322+0000 7ff04d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:38.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.654+0000 7ff056f6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:38.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.654+0000 7ff056f6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:38.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.654+0000 7ff056f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:09:38.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.654+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff030005330 con 0x7ff0500a9d70 2024-03-29T11:09:38.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.657+0000 7ff04d7fa640 1 -- 172.21.15.105:0/2412702720 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff0440523f0 con 0x7ff0500a9d70 2024-03-29T11:09:38.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.657+0000 7ff04d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:38.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.657+0000 7ff04d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:38.638 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:38.638 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:38.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.663+0000 7ff056f6b640 10 librados: watch_flush enter 2024-03-29T11:09:38.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.663+0000 7ff056f6b640 10 librados: watch_flush exit 2024-03-29T11:09:38.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.663+0000 7ff056f6b640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:09:38.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.663+0000 7ff056f6b640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff03406df50 legacy=0x7ff034070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 10 monclient: shutdown 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 >> v1:172.21.15.105:6789/0 conn(0x7ff0500a9d70 legacy=0x7ff0501da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 shutdown_connections 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 >> 172.21.15.105:0/2412702720 conn(0x7ff050066a00 msgr2=0x7ff05013e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:38.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.664+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 shutdown_connections 2024-03-29T11:09:38.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.665+0000 7ff056f6b640 1 -- 172.21.15.105:0/2412702720 wait complete. 2024-03-29T11:09:38.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:38.665+0000 7ff056f6b640 1 librados: shutdown 2024-03-29T11:09:38.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:45.472 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.647+0000 7fbfad04e640 10 monclient: get_monmap_and_config 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.647+0000 7fbfad04e640 10 monclient: build_initial_monmap 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.647+0000 7fbfad04e640 10 monclient: monmap: 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:45.626 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:45.648580+0000 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:45.648580+0000 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:45.627 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:45.628 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.648+0000 7fbfad04e640 1 Processor -- start 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.648+0000 7fbfad04e640 1 -- start start 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.648+0000 7fbfad04e640 10 monclient: init 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-c con 0x7fbfa81457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-b con 0x7fbfa813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-a con 0x7fbfa80a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80aaef0 con 0x7fbfa80a9d70 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8145bd0 con 0x7fbfa81457f0 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8145eb0 con 0x7fbfa813f650 2024-03-29T11:09:45.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfad04e640 10 monclient(hunting): authenticate will time out at 859.925378s 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfa6d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbfa81457f0 0x7fbfa813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:36134/0 (socket says 172.21.15.105:36134) 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.649+0000 7fbfa6d9d640 1 -- 172.21.15.105:0/2543410919 learned_addr learned my addr 172.21.15.105:0/2543410919 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066574230 0 0) 0x7fbfa8145bd0 con 0x7fbfa81457f0 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:09:45.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90003520 con 0x7fbfa81457f0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086307921 0 0) 0x7fbf90003520 con 0x7fbfa81457f0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 >> v1:172.21.15.105:6790/0 conn(0x7fbfa813f650 legacy=0x7fbfa80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 >> v1:172.21.15.105:6789/0 conn(0x7fbfa80a9d70 legacy=0x7fbfa814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8146340 con 0x7fbfa81457f0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:45.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbf98001690 con 0x7fbfa81457f0 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfa5d9b640 10 monclient: dump: 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:45.633 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfad04e640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.650+0000 7fbfad04e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf980017f0 con 0x7fbfa81457f0 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 1 -- 172.21.15.105:0/2543410919 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbf98001c40 con 0x7fbfa81457f0 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfa5d9b640 10 monclient: dump: 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.634 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfad04e640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:45.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.651+0000 7fbfad04e640 4 set_mon_vals no callback set 2024-03-29T11:09:45.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 10 monclient: shutdown 2024-03-29T11:09:45.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 1 -- 172.21.15.105:0/2543410919 >> v1:172.21.15.110:6789/0 conn(0x7fbfa81457f0 legacy=0x7fbfa813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 1 -- 172.21.15.105:0/2543410919 shutdown_connections 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 1 -- 172.21.15.105:0/2543410919 >> 172.21.15.105:0/2543410919 conn(0x7fbfa8066a00 msgr2=0x7fbfa8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.652+0000 7fbfad04e640 1 -- 172.21.15.105:0/2543410919 shutdown_connections 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.653+0000 7fbfad04e640 1 -- 172.21.15.105:0/2543410919 wait complete. 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.653+0000 7fbfad04e640 10 monclient: build_initial_monmap 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 10 monclient: monmap: 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:45.644 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 librados: starting msgr at 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 librados: starting objecter 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 Processor -- start 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 -- start start 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 librados: setting wanted keys 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 1 librados: calling monclient init 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.654+0000 7fbfad04e640 10 monclient: init 2024-03-29T11:09:45.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.655+0000 7fbfad04e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.655+0000 7fbfad04e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-a con 0x7fbfa80a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-c con 0x7fbfa813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 10 monclient(hunting): picked mon.noname-b con 0x7fbfa81457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8141970 con 0x7fbfa80a9d70 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8139830 con 0x7fbfa81457f0 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa813fda0 con 0x7fbfa813f650 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfad04e640 10 monclient(hunting): authenticate will time out at 859.932161s 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfa759e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbfa81457f0 0x7fbfa81e2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:36504/0 (socket says 172.21.15.105:36504) 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbfa759e640 1 -- 172.21.15.105:0/2940565122 learned_addr learned my addr 172.21.15.105:0/2940565122 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbfa81457f0 2024-03-29T11:09:45.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbfa813f650 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbfa80a9d70 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.656+0000 7fbf8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721948124 0 0) 0x7fbfa8139830 con 0x7fbfa81457f0 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf94004eb0 con 0x7fbfa81457f0 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.657+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735212320 0 0) 0x7fbfa8141970 con 0x7fbfa80a9d70 2024-03-29T11:09:45.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa8139830 con 0x7fbfa80a9d70 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347575832 0 0) 0x7fbfa813fda0 con 0x7fbfa813f650 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa8141970 con 0x7fbfa813f650 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54179784 0 0) 0x7fbf94004eb0 con 0x7fbfa81457f0 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfa813fda0 con 0x7fbfa81457f0 2024-03-29T11:09:45.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbfa0001c50 con 0x7fbfa81457f0 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:45.649 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.658+0000 7fbf8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.659+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3226072208 0 0) 0x7fbfa813fda0 con 0x7fbfa81457f0 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.659+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 >> v1:172.21.15.105:6790/0 conn(0x7fbfa813f650 legacy=0x7fbfa81daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.659+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 >> v1:172.21.15.105:6789/0 conn(0x7fbfa80a9d70 legacy=0x7fbfa8140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.659+0000 7fbf8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.659+0000 7fbf8f7fe640 1 monclient: found mon.b 2024-03-29T11:09:45.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8140300 con 0x7fbfa81457f0 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 10 monclient: _renew_subs 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa8141500 con 0x7fbfa81457f0 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 10 monclient: _renew_subs 2024-03-29T11:09:45.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81db7f0 con 0x7fbfa81457f0 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfad04e640 1 librados: init done 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa0003990 con 0x7fbfa81457f0 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbfa0003de0 con 0x7fbfa81457f0 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbfa559a640 4 set_mon_vals no callback set 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:45.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.660+0000 7fbf8f7fe640 10 monclient: dump: 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:45.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.661+0000 7fbfad04e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.662+0000 7fbfad04e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.662+0000 7fbfad04e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.662+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf88005080 con 0x7fbfa81457f0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.663+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fbfa0003f40 con 0x7fbfa81457f0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbfa6d9d640 10 monclient: get_auth_request con 0x7fbf9406dc70 auth_method 0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fbfa0084d40 con 0x7fbfa81457f0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 10 client.4282.objecter ms_dispatch 0x7fbfa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:45.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:09:45.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.664+0000 7fbf8f7fe640 10 client.4282.objecter ms_handle_connect 0x7fbf9406dc70 2024-03-29T11:09:45.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.670+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfa0050620 con 0x7fbfa81457f0 2024-03-29T11:09:45.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.670+0000 7fbf8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:45.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:45.670+0000 7fbf8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:46.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.084+0000 7fbfad04e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:46.063 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.084+0000 7fbfad04e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:46.063 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.084+0000 7fbfad04e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:09:46.063 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.084+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf88005330 con 0x7fbfa81457f0 2024-03-29T11:09:46.063 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.085+0000 7fbf8f7fe640 1 -- 172.21.15.105:0/2940565122 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fbfa0052550 con 0x7fbfa81457f0 2024-03-29T11:09:46.063 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.085+0000 7fbf8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:46.064 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.085+0000 7fbf8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:46.064 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:46.064 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.089+0000 7fbfad04e640 10 librados: watch_flush enter 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.089+0000 7fbfad04e640 10 librados: watch_flush exit 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.089+0000 7fbfad04e640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.089+0000 7fbfad04e640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 >> v1:172.21.15.110:6816/1087970004 conn(0x7fbf9406dc70 legacy=0x7fbf94070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 10 monclient: shutdown 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:46.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 >> v1:172.21.15.110:6789/0 conn(0x7fbfa81457f0 legacy=0x7fbfa81e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:46.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 shutdown_connections 2024-03-29T11:09:46.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 >> 172.21.15.105:0/2940565122 conn(0x7fbfa8066a00 msgr2=0x7fbfa80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:46.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.090+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 shutdown_connections 2024-03-29T11:09:46.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.091+0000 7fbfad04e640 1 -- 172.21.15.105:0/2940565122 wait complete. 2024-03-29T11:09:46.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:46.091+0000 7fbfad04e640 1 librados: shutdown 2024-03-29T11:09:46.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:52.890 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.040+0000 7f975f53b640 10 monclient: get_monmap_and_config 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.040+0000 7f975f53b640 10 monclient: build_initial_monmap 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.040+0000 7f975f53b640 10 monclient: monmap: 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:09:53.041979+0000 2024-03-29T11:09:53.019 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:09:53.041979+0000 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:09:53.020 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.041+0000 7f975f53b640 1 Processor -- start 2024-03-29T11:09:53.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.042+0000 7f975f53b640 1 -- start start 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.042+0000 7f975f53b640 10 monclient: init 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-a con 0x7f97581457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-b con 0x7f975813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-c con 0x7f97580a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580aaef0 con 0x7f97581457f0 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975cad6640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f975813f650 0x7f97580b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:33190/0 (socket says 172.21.15.105:33190) 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975cad6640 1 -- 172.21.15.105:0/948210544 learned_addr learned my addr 172.21.15.105:0/948210544 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758145bd0 con 0x7f97580a9d70 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758145eb0 con 0x7f975813f650 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:53.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.043+0000 7f975f53b640 10 monclient(hunting): authenticate will time out at 867.319527s 2024-03-29T11:09:53.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787908566 0 0) 0x7f9758145eb0 con 0x7f975813f650 2024-03-29T11:09:53.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:53.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:53.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:53.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:09:53.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9744003520 con 0x7f975813f650 2024-03-29T11:09:53.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959810033 0 0) 0x7f9744003520 con 0x7f975813f650 2024-03-29T11:09:53.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 >> v1:172.21.15.110:6789/0 conn(0x7f97580a9d70 legacy=0x7f975814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:53.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.044+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 >> v1:172.21.15.105:6789/0 conn(0x7f97581457f0 legacy=0x7f975813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 1 monclient: found mon.noname-b 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758146340 con 0x7f975813f650 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: _finish_auth 0 2024-03-29T11:09:53.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9748001690 con 0x7f975813f650 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 10 monclient: dump: 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:53.028 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f975f53b640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f975f53b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.046+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9748001920 con 0x7f975813f650 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f974ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f974ffff640 1 -- 172.21.15.105:0/948210544 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9748001cb0 con 0x7f975813f650 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f974ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f974ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f974ffff640 10 monclient: dump: 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:53.029 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f975f53b640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f975f53b640 4 set_mon_vals no callback set 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f975f53b640 10 monclient: shutdown 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f975f53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.047+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 >> v1:172.21.15.105:6790/0 conn(0x7f975813f650 legacy=0x7f97580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 shutdown_connections 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 >> 172.21.15.105:0/948210544 conn(0x7f9758066a00 msgr2=0x7f9758066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 shutdown_connections 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 1 -- 172.21.15.105:0/948210544 wait complete. 2024-03-29T11:09:53.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 10 monclient: build_initial_monmap 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.048+0000 7f975f53b640 10 monclient: monmap: 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:09:53.031 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 librados: starting msgr at 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 librados: starting objecter 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 Processor -- start 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 -- start start 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 librados: setting wanted keys 2024-03-29T11:09:53.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 1 librados: calling monclient init 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.049+0000 7f975f53b640 10 monclient: init 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.050+0000 7f975f53b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.050+0000 7f975f53b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.050+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-a con 0x7f97580a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.050+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-b con 0x7f975813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 10 monclient(hunting): picked mon.noname-c con 0x7f97581457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758071d90 con 0x7f97580a9d70 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975813c0d0 con 0x7f975813f650 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580ac2d0 con 0x7f97581457f0 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975f53b640 10 monclient(hunting): authenticate will time out at 867.327333s 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975dad8640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f97581457f0 0x7f97581e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:33724/0 (socket says 172.21.15.105:33724) 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f975dad8640 1 -- 172.21.15.105:0/1685728596 learned_addr learned my addr 172.21.15.105:0/1685728596 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:09:53.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f974dffb640 10 client.?.objecter ms_handle_connect 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.051+0000 7f974dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125975300 0 0) 0x7f97580ac2d0 con 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417743715 0 0) 0x7f973c004eb0 con 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97580ac2d0 con 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9750001c60 con 0x7f97581457f0 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:53.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): dump: 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.052+0000 7f974dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4139332561 0 0) 0x7f97580ac2d0 con 0x7f97581457f0 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 >> v1:172.21.15.110:6789/0 conn(0x7f975813f650 legacy=0x7f975806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:53.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 >> v1:172.21.15.105:6789/0 conn(0x7f97580a9d70 legacy=0x7f975806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 monclient: found mon.c 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580ac5b0 con 0x7f97581457f0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f975f53b640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f975f53b640 10 monclient: _renew_subs 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9750003970 con 0x7f97581457f0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f975f53b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975806f240 con 0x7f97581457f0 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:53.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9750003d80 con 0x7f97581457f0 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974dffb640 10 monclient: dump: 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.053+0000 7f974f7fe640 4 set_mon_vals no callback set 2024-03-29T11:09:53.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.054+0000 7f975f53b640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:53.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.054+0000 7f975f53b640 10 monclient: _renew_subs 2024-03-29T11:09:53.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.054+0000 7f975f53b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.054+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97580b14d0 con 0x7f97581457f0 2024-03-29T11:09:53.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.054+0000 7f975f53b640 1 librados: init done 2024-03-29T11:09:53.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.055+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f9750003ee0 con 0x7f97581457f0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f975f53b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f975f53b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f975f53b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975813bf40 con 0x7f97581457f0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f9750084e10 con 0x7f97581457f0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 10 client.4301.objecter ms_dispatch 0x7f9758000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f975d2d7640 10 monclient: get_auth_request con 0x7f973c06dbd0 auth_method 0 2024-03-29T11:09:53.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.056+0000 7f974dffb640 10 client.4301.objecter ms_handle_connect 0x7f973c06dbd0 2024-03-29T11:09:53.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.064+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97500506f0 con 0x7f97581457f0 2024-03-29T11:09:53.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.064+0000 7f974dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:53.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.064+0000 7f974dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:53.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.369+0000 7f975f53b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:53.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.369+0000 7f975f53b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:53.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.369+0000 7f975f53b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:09:53.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.369+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97580b1880 con 0x7f97581457f0 2024-03-29T11:09:53.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.370+0000 7f974dffb640 1 -- 172.21.15.105:0/1685728596 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f9750052620 con 0x7f97581457f0 2024-03-29T11:09:53.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.370+0000 7f974dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:53.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.370+0000 7f974dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:53.349 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:09:53.349 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.373+0000 7f975f53b640 10 librados: watch_flush enter 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.373+0000 7f975f53b640 10 librados: watch_flush exit 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.373+0000 7f975f53b640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.373+0000 7f975f53b640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 >> v1:172.21.15.110:6816/1087970004 conn(0x7f973c06dbd0 legacy=0x7f973c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 10 monclient: shutdown 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 >> v1:172.21.15.105:6790/0 conn(0x7f97581457f0 legacy=0x7f97581e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 shutdown_connections 2024-03-29T11:09:53.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.374+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 >> 172.21.15.105:0/1685728596 conn(0x7f9758066a00 msgr2=0x7f97580b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:53.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.375+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 shutdown_connections 2024-03-29T11:09:53.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.375+0000 7f975f53b640 1 -- 172.21.15.105:0/1685728596 wait complete. 2024-03-29T11:09:53.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:09:53.375+0000 7f975f53b640 1 librados: shutdown 2024-03-29T11:09:53.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:00.175 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:00.313 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.334+0000 7f28acf90640 10 monclient: get_monmap_and_config 2024-03-29T11:10:00.313 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.334+0000 7f28acf90640 10 monclient: build_initial_monmap 2024-03-29T11:10:00.313 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.334+0000 7f28acf90640 10 monclient: monmap: 2024-03-29T11:10:00.313 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:00.313 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:00.335934+0000 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:00.335934+0000 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:00.314 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.315 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.335+0000 7f28acf90640 1 Processor -- start 2024-03-29T11:10:00.315 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.335+0000 7f28acf90640 1 -- start start 2024-03-29T11:10:00.315 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.335+0000 7f28acf90640 10 monclient: init 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-a con 0x7f28a81457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-c con 0x7f28a813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-b con 0x7f28a80a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80aaef0 con 0x7f28a81457f0 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8145bd0 con 0x7f28a813f650 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8145eb0 con 0x7f28a80a9d70 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28acf90640 10 monclient(hunting): authenticate will time out at 874.613908s 2024-03-29T11:10:00.317 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28a6d9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f28a80a9d70 0x7f28a814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:59208/0 (socket says 172.21.15.105:59208) 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.337+0000 7f28a6d9d640 1 -- 172.21.15.105:0/375461627 learned_addr learned my addr 172.21.15.105:0/375461627 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809770207 0 0) 0x7f28a8145eb0 con 0x7f28a80a9d70 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890003520 con 0x7f28a80a9d70 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774904376 0 0) 0x7f28a8145bd0 con 0x7f28a813f650 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:00.318 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:10:00.319 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a8145eb0 con 0x7f28a813f650 2024-03-29T11:10:00.319 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783934643 0 0) 0x7f2890003520 con 0x7f28a80a9d70 2024-03-29T11:10:00.319 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 >> v1:172.21.15.110:6789/0 conn(0x7f28a813f650 legacy=0x7f28a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.319 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.338+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 >> v1:172.21.15.105:6789/0 conn(0x7f28a81457f0 legacy=0x7f28a813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:00.323 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.339+0000 7f28a559a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:00.323 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.339+0000 7f28a559a640 1 monclient: found mon.noname-b 2024-03-29T11:10:00.323 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.339+0000 7f28a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.339+0000 7f28a559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.339+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8146340 con 0x7f28a80a9d70 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: _finish_auth 0 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f28a0001a10 con 0x7f28a80a9d70 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28acf90640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28acf90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: dump: 2024-03-29T11:10:00.324 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28a0001c30 con 0x7f28a80a9d70 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28acf90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:00.325 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28a559a640 1 -- 172.21.15.105:0/375461627 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f28a0001fc0 con 0x7f28a80a9d70 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28acf90640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.340+0000 7f28acf90640 4 set_mon_vals no callback set 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28a559a640 10 monclient: dump: 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.326 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28acf90640 10 monclient: shutdown 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28acf90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28acf90640 1 -- 172.21.15.105:0/375461627 >> v1:172.21.15.105:6790/0 conn(0x7f28a80a9d70 legacy=0x7f28a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28acf90640 1 -- 172.21.15.105:0/375461627 shutdown_connections 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.341+0000 7f28acf90640 1 -- 172.21.15.105:0/375461627 >> 172.21.15.105:0/375461627 conn(0x7f28a8066a00 msgr2=0x7f28a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.342+0000 7f28acf90640 1 -- 172.21.15.105:0/375461627 shutdown_connections 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.342+0000 7f28acf90640 1 -- 172.21.15.105:0/375461627 wait complete. 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 10 monclient: build_initial_monmap 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 10 monclient: monmap: 2024-03-29T11:10:00.327 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 librados: starting msgr at 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 librados: starting objecter 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 Processor -- start 2024-03-29T11:10:00.328 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 -- start start 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 librados: setting wanted keys 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 1 librados: calling monclient init 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.343+0000 7f28acf90640 10 monclient: init 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.344+0000 7f28acf90640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.344+0000 7f28acf90640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.344+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-c con 0x7f28a80a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.344+0000 7f28a659c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f28a80a9d70 0x7f28a81d7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60292/0 (socket says 172.21.15.105:60292) 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.345+0000 7f28a659c640 1 -- 172.21.15.105:0/1901813810 learned_addr learned my addr 172.21.15.105:0/1901813810 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.345+0000 7f2896ffd640 10 client.?.objecter ms_handle_connect 0x7f28a80a9d70 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.345+0000 7f2896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:00.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.345+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-b con 0x7f28a813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 10 monclient(hunting): picked mon.noname-a con 0x7f28a81457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a81d4c30 con 0x7f28a81457f0 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a81d7f10 con 0x7f28a813f650 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8144810 con 0x7f28a80a9d70 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.346+0000 7f28acf90640 10 monclient(hunting): authenticate will time out at 874.622554s 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 client.?.objecter ms_handle_connect 0x7f28a813f650 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962037527 0 0) 0x7f28a81d7f10 con 0x7f28a813f650 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:00.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888004eb0 con 0x7f28a813f650 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701825002 0 0) 0x7f28a8144810 con 0x7f28a80a9d70 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a81d7f10 con 0x7f28a80a9d70 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 client.?.objecter ms_handle_connect 0x7f28a81457f0 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986351617 0 0) 0x7f28a81d4c30 con 0x7f28a81457f0 2024-03-29T11:10:00.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a8144810 con 0x7f28a81457f0 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932913965 0 0) 0x7f2888004eb0 con 0x7f28a813f650 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a81d4c30 con 0x7f28a813f650 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f289c001980 con 0x7f28a813f650 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:00.337 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.347+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376658132 0 0) 0x7f28a8144810 con 0x7f28a81457f0 2024-03-29T11:10:00.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2888004eb0 con 0x7f28a81457f0 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f28a0001da0 con 0x7f28a81457f0 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:00.339 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 258111430 0 0) 0x7f2888004eb0 con 0x7f28a81457f0 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 >> v1:172.21.15.105:6790/0 conn(0x7f28a80a9d70 legacy=0x7f28a81d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 >> v1:172.21.15.110:6789/0 conn(0x7f28a813f650 legacy=0x7f28a80b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 monclient: found mon.a 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a81449a0 con 0x7f28a81457f0 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:00.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 10 monclient: _renew_subs 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a80b1480 con 0x7f28a81457f0 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28a0004cc0 con 0x7f28a81457f0 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f28a0004fd0 con 0x7f28a81457f0 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f2896ffd640 10 monclient: dump: 2024-03-29T11:10:00.341 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28a4d99640 4 set_mon_vals no callback set 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 10 monclient: _renew_subs 2024-03-29T11:10:00.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a8140750 con 0x7f28a81457f0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.348+0000 7f28acf90640 1 librados: init done 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.350+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f28a001b020 con 0x7f28a81457f0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f28a0084e40 con 0x7f28a81457f0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f2896ffd640 10 client.4278.objecter ms_dispatch 0x7f28a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f2896ffd640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f2896ffd640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f28acf90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f28acf90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f28acf90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.351+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a81d7d10 con 0x7f28a81457f0 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.352+0000 7f2896ffd640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:10:00.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.352+0000 7f28a659c640 10 monclient: get_auth_request con 0x7f288806e060 auth_method 0 2024-03-29T11:10:00.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.352+0000 7f2896ffd640 10 client.4278.objecter ms_handle_connect 0x7f288806e060 2024-03-29T11:10:00.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.360+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28a0050720 con 0x7f28a81457f0 2024-03-29T11:10:00.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.360+0000 7f2896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:00.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.360+0000 7f2896ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:00.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.657+0000 7f28acf90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.657+0000 7f28acf90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.657+0000 7f28acf90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.657+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28a8075f10 con 0x7f28a81457f0 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.658+0000 7f2896ffd640 1 -- 172.21.15.105:0/1901813810 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f28a0052650 con 0x7f28a81457f0 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.658+0000 7f2896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:00.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.658+0000 7f2896ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:00.637 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:00.637 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:00.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 10 librados: watch_flush enter 2024-03-29T11:10:00.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 10 librados: watch_flush exit 2024-03-29T11:10:00.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 >> v1:172.21.15.110:6816/1087970004 conn(0x7f288806e060 legacy=0x7f2888070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 10 monclient: shutdown 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 >> v1:172.21.15.105:6789/0 conn(0x7f28a81457f0 legacy=0x7f28a8140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 shutdown_connections 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.661+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 >> 172.21.15.105:0/1901813810 conn(0x7f28a8066a00 msgr2=0x7f28a813b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:00.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.662+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 shutdown_connections 2024-03-29T11:10:00.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.662+0000 7f28acf90640 1 -- 172.21.15.105:0/1901813810 wait complete. 2024-03-29T11:10:00.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:00.662+0000 7f28acf90640 1 librados: shutdown 2024-03-29T11:10:00.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:07.463 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:07.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.638+0000 7fe1bdb99640 10 monclient: get_monmap_and_config 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.638+0000 7fe1bdb99640 10 monclient: build_initial_monmap 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.638+0000 7fe1bdb99640 10 monclient: monmap: 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:07.639514+0000 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:07.639514+0000 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:07.617 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.641+0000 7fe1bdb99640 1 Processor -- start 2024-03-29T11:10:07.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.641+0000 7fe1bdb99640 1 -- start start 2024-03-29T11:10:07.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.641+0000 7fe1bdb99640 10 monclient: init 2024-03-29T11:10:07.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.642+0000 7fe1bdb99640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:07.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.642+0000 7fe1bdb99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-b con 0x7fe1b80b2800 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-a con 0x7fe1b8142b20 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b813c040 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80aad10 con 0x7fe1b8142b20 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b8141f30 con 0x7fe1b813c040 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80aaf60 con 0x7fe1b80b2800 2024-03-29T11:10:07.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:07.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bdb99640 10 monclient(hunting): authenticate will time out at 881.921484s 2024-03-29T11:10:07.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bcb97640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fe1b80b2800 0x7fe1b8142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:49085/0 (socket says 172.21.15.105:49085) 2024-03-29T11:10:07.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.645+0000 7fe1bcb97640 1 -- 172.21.15.105:0/2878647237 learned_addr learned my addr 172.21.15.105:0/2878647237 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:07.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872271455 0 0) 0x7fe1b80aaf60 con 0x7fe1b80b2800 2024-03-29T11:10:07.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:07.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a4003520 con 0x7fe1b80b2800 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991518532 0 0) 0x7fe1b80aad10 con 0x7fe1b8142b20 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b80aaf60 con 0x7fe1b8142b20 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289905052 0 0) 0x7fe1a4003520 con 0x7fe1b80b2800 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 >> v1:172.21.15.110:6789/0 conn(0x7fe1b813c040 legacy=0x7fe1b814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 >> v1:172.21.15.105:6789/0 conn(0x7fe1b8142b20 legacy=0x7fe1b813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:07.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 monclient: found mon.noname-b 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80b3900 con 0x7fe1b80b2800 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe1a8001a10 con 0x7fe1b80b2800 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.649+0000 7fe1b77fe640 10 monclient: dump: 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.630 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1a8001b70 con 0x7fe1b80b2800 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 1 -- 172.21.15.105:0/2878647237 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe1a8001f80 con 0x7fe1b80b2800 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1b77fe640 10 monclient: dump: 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:07.631 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:07.632 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1bdb99640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1bdb99640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.650+0000 7fe1bdb99640 4 set_mon_vals no callback set 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.651+0000 7fe1bdb99640 10 monclient: shutdown 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.651+0000 7fe1bdb99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.651+0000 7fe1bdb99640 1 -- 172.21.15.105:0/2878647237 >> v1:172.21.15.105:6790/0 conn(0x7fe1b80b2800 legacy=0x7fe1b8142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.652+0000 7fe1bdb99640 1 -- 172.21.15.105:0/2878647237 shutdown_connections 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.652+0000 7fe1bdb99640 1 -- 172.21.15.105:0/2878647237 >> 172.21.15.105:0/2878647237 conn(0x7fe1b806d0e0 msgr2=0x7fe1b8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.652+0000 7fe1bdb99640 1 -- 172.21.15.105:0/2878647237 shutdown_connections 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.652+0000 7fe1bdb99640 1 -- 172.21.15.105:0/2878647237 wait complete. 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.653+0000 7fe1bdb99640 10 monclient: build_initial_monmap 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.653+0000 7fe1bdb99640 10 monclient: monmap: 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:07.635 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.653+0000 7fe1bdb99640 1 librados: starting msgr at 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.653+0000 7fe1bdb99640 1 librados: starting objecter 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.654+0000 7fe1bdb99640 1 Processor -- start 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.654+0000 7fe1bdb99640 1 -- start start 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.654+0000 7fe1bdb99640 1 librados: setting wanted keys 2024-03-29T11:10:07.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.654+0000 7fe1bdb99640 1 librados: calling monclient init 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.654+0000 7fe1bdb99640 10 monclient: init 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.655+0000 7fe1bdb99640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.655+0000 7fe1bdb99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-b con 0x7fe1b80b2800 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-a con 0x7fe1b813c040 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b8142b20 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b813f150 con 0x7fe1b813c040 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b81d28a0 con 0x7fe1b80b2800 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b813ca80 con 0x7fe1b8142b20 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:07.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.656+0000 7fe1bdb99640 10 monclient(hunting): authenticate will time out at 881.932414s 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1bcb97640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe1b80b2800 0x7fe1b81d2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:34837/0 (socket says 172.21.15.105:34837) 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1bcb97640 1 -- 172.21.15.105:0/1738592053 learned_addr learned my addr 172.21.15.105:0/1738592053 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 client.?.objecter ms_handle_connect 0x7fe1b80b2800 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913624455 0 0) 0x7fe1b81d28a0 con 0x7fe1b80b2800 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.662+0000 7fe1b57fa640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:10:07.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe198004eb0 con 0x7fe1b80b2800 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956857176 0 0) 0x7fe198004eb0 con 0x7fe1b80b2800 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1b81d28a0 con 0x7fe1b80b2800 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe1a8001fc0 con 0x7fe1b80b2800 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 10 monclient(hunting): dump: 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:07.643 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.663+0000 7fe1b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4096754406 0 0) 0x7fe1b81d28a0 con 0x7fe1b80b2800 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 >> v1:172.21.15.105:6790/0 conn(0x7fe1b8142b20 legacy=0x7fe1b813d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 >> v1:172.21.15.105:6789/0 conn(0x7fe1b813c040 legacy=0x7fe1b8132c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 1 monclient: found mon.b 2024-03-29T11:10:07.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8133df0 con 0x7fe1b80b2800 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 10 monclient: _finish_auth 0 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 10 monclient: _renew_subs 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b81d34d0 con 0x7fe1b80b2800 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 10 monclient: _renew_subs 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b8133550 con 0x7fe1b80b2800 2024-03-29T11:10:07.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.664+0000 7fe1bdb99640 1 librados: init done 2024-03-29T11:10:07.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1a8004e10 con 0x7fe1b80b2800 2024-03-29T11:10:07.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:07.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe1a8005220 con 0x7fe1b80b2800 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b57fa640 10 monclient: dump: 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:07.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1bdb99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:07.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1bdb99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:07.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1bdb99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:10:07.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b804f730 con 0x7fe1b80b2800 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.666+0000 7fe1b67fc640 4 set_mon_vals no callback set 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fe1a8005440 con 0x7fe1b80b2800 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fe1a80850b0 con 0x7fe1b80b2800 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 10 client.4291.objecter ms_dispatch 0x7fe1b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.669+0000 7fe1b57fa640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.670+0000 7fe1b7fff640 10 monclient: get_auth_request con 0x7fe19806dc90 auth_method 0 2024-03-29T11:10:07.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.671+0000 7fe1b57fa640 10 client.4291.objecter ms_handle_connect 0x7fe19806dc90 2024-03-29T11:10:07.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.675+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1a8050990 con 0x7fe1b80b2800 2024-03-29T11:10:07.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.675+0000 7fe1b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:07.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:07.675+0000 7fe1b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:07.988 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.009+0000 7fe1bdb99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:07.988 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.009+0000 7fe1bdb99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:07.988 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.009+0000 7fe1bdb99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:10:07.988 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.009+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b8074d80 con 0x7fe1b80b2800 2024-03-29T11:10:07.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.012+0000 7fe1b57fa640 1 -- 172.21.15.105:0/1738592053 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fe1a80528c0 con 0x7fe1b80b2800 2024-03-29T11:10:07.991 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.012+0000 7fe1b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:07.991 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.012+0000 7fe1b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:07.992 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:07.992 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:07.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.017+0000 7fe1bdb99640 10 librados: watch_flush enter 2024-03-29T11:10:07.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.017+0000 7fe1bdb99640 10 librados: watch_flush exit 2024-03-29T11:10:07.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.017+0000 7fe1bdb99640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:10:07.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.017+0000 7fe1bdb99640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:10:07.998 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.019+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 >> v1:172.21.15.110:6816/1087970004 conn(0x7fe19806dc90 legacy=0x7fe198070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:07.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.019+0000 7fe1bdb99640 10 monclient: shutdown 2024-03-29T11:10:07.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.019+0000 7fe1bdb99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:07.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.019+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 >> v1:172.21.15.110:6789/0 conn(0x7fe1b80b2800 legacy=0x7fe1b81d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:08.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.021+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 shutdown_connections 2024-03-29T11:10:08.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.021+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 >> 172.21.15.105:0/1738592053 conn(0x7fe1b806d0e0 msgr2=0x7fe1b8139710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:08.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.021+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 shutdown_connections 2024-03-29T11:10:08.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.021+0000 7fe1bdb99640 1 -- 172.21.15.105:0/1738592053 wait complete. 2024-03-29T11:10:08.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:08.021+0000 7fe1bdb99640 1 librados: shutdown 2024-03-29T11:10:08.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:14.825 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:14.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.006+0000 7f25117c8640 10 monclient: get_monmap_and_config 2024-03-29T11:10:14.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.006+0000 7f25117c8640 10 monclient: build_initial_monmap 2024-03-29T11:10:14.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.006+0000 7f25117c8640 10 monclient: monmap: 2024-03-29T11:10:14.984 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:15.007261+0000 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:15.007261+0000 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:14.985 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:14.988 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.010+0000 7f25117c8640 1 Processor -- start 2024-03-29T11:10:14.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 1 -- start start 2024-03-29T11:10:14.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient: init 2024-03-29T11:10:14.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-a con 0x7f250c1457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-b con 0x7f250c13f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-c con 0x7f250c0a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0aaef0 con 0x7f250c1457f0 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c145bd0 con 0x7f250c0a9d70 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c145eb0 con 0x7f250c13f650 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.012+0000 7f25117c8640 10 monclient(hunting): authenticate will time out at 889.288946s 2024-03-29T11:10:14.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.015+0000 7f250b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f250c0a9d70 0x7f250c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:52765/0 (socket says 172.21.15.105:52765) 2024-03-29T11:10:14.994 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.015+0000 7f250b7fe640 1 -- 172.21.15.105:0/4212817240 learned_addr learned my addr 172.21.15.105:0/4212817240 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:14.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764965266 0 0) 0x7f250c145eb0 con 0x7f250c13f650 2024-03-29T11:10:14.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f4003520 con 0x7f250c13f650 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750070215 0 0) 0x7f24f4003520 con 0x7f250c13f650 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 >> v1:172.21.15.110:6789/0 conn(0x7f250c0a9d70 legacy=0x7f250c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:14.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.017+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 >> v1:172.21.15.105:6789/0 conn(0x7f250c1457f0 legacy=0x7f250c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 1 monclient: found mon.noname-b 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c146340 con 0x7f250c13f650 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: _finish_auth 0 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f24fc001690 con 0x7f250c13f650 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:14.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f2509ffb640 10 monclient: dump: 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f25117c8640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:10:15.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.020+0000 7f25117c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.021+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24fc0018b0 con 0x7f250c13f650 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.021+0000 7f2509ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.021+0000 7f25117c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.021+0000 7f25117c8640 4 set_mon_vals no callback set 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f2509ffb640 1 -- 172.21.15.105:0/4212817240 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f24fc001cc0 con 0x7f250c13f650 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f2509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f2509ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f2509ffb640 10 monclient: dump: 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.002 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f25117c8640 10 monclient: shutdown 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f25117c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f25117c8640 1 -- 172.21.15.105:0/4212817240 >> v1:172.21.15.105:6790/0 conn(0x7f250c13f650 legacy=0x7f250c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f25117c8640 1 -- 172.21.15.105:0/4212817240 shutdown_connections 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.022+0000 7f25117c8640 1 -- 172.21.15.105:0/4212817240 >> 172.21.15.105:0/4212817240 conn(0x7f250c066a00 msgr2=0x7f250c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.023+0000 7f25117c8640 1 -- 172.21.15.105:0/4212817240 shutdown_connections 2024-03-29T11:10:15.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.023+0000 7f25117c8640 1 -- 172.21.15.105:0/4212817240 wait complete. 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.024+0000 7f25117c8640 10 monclient: build_initial_monmap 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.024+0000 7f25117c8640 10 monclient: monmap: 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:15.004 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.024+0000 7f25117c8640 1 librados: starting msgr at 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.024+0000 7f25117c8640 1 librados: starting objecter 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.024+0000 7f25117c8640 1 Processor -- start 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 1 -- start start 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 1 librados: setting wanted keys 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 1 librados: calling monclient init 2024-03-29T11:10:15.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 10 monclient: init 2024-03-29T11:10:15.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:15.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.025+0000 7f25117c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:10:15.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.028+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-c con 0x7f250c0a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:15.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.028+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-b con 0x7f250c1457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:15.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f250affd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f250c0a9d70 0x7f250c1e7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55013/0 (socket says 172.21.15.105:55013) 2024-03-29T11:10:15.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f250affd640 1 -- 172.21.15.105:0/1481234078 learned_addr learned my addr 172.21.15.105:0/1481234078 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:15.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 10 monclient(hunting): picked mon.noname-a con 0x7f250c06e5c0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:15.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0700f0 con 0x7f250c06e5c0 2024-03-29T11:10:15.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0703d0 con 0x7f250c1457f0 2024-03-29T11:10:15.011 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c06e0a0 con 0x7f250c0a9d70 2024-03-29T11:10:15.011 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:15.011 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.032+0000 7f25117c8640 10 monclient(hunting): authenticate will time out at 889.308405s 2024-03-29T11:10:15.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 client.?.objecter ms_handle_connect 0x7f250c0a9d70 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 client.?.objecter ms_handle_connect 0x7f250c06e5c0 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783695670 0 0) 0x7f250c06e0a0 con 0x7f250c0a9d70 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f8004eb0 con 0x7f250c0a9d70 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127801551 0 0) 0x7f250c0700f0 con 0x7f250c06e5c0 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:15.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.033+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c06e0a0 con 0x7f250c06e5c0 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904884501 0 0) 0x7f24f8004eb0 con 0x7f250c0a9d70 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f250c0700f0 con 0x7f250c0a9d70 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2504001cb0 con 0x7f250c0a9d70 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 10 monclient(hunting): dump: 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:15.016 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.035+0000 7f24f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 client.?.objecter ms_handle_connect 0x7f250c1457f0 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 404269707 0 0) 0x7f250c0700f0 con 0x7f250c0a9d70 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 >> v1:172.21.15.110:6789/0 conn(0x7f250c1457f0 legacy=0x7f250c141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 >> v1:172.21.15.105:6789/0 conn(0x7f250c06e5c0 legacy=0x7f250c1ef780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 1 monclient: found mon.c 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c1e8200 con 0x7f250c0a9d70 2024-03-29T11:10:15.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f24f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 10 monclient: _renew_subs 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c141d80 con 0x7f250c0a9d70 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 10 monclient: _renew_subs 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c06f090 con 0x7f250c0a9d70 2024-03-29T11:10:15.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.036+0000 7f25117c8640 1 librados: init done 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.037+0000 7f24f17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.037+0000 7f24f17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.037+0000 7f24f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.037+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c06e230 con 0x7f250c0a9d70 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.038+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2504001e10 con 0x7f250c0a9d70 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.038+0000 7f24f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.039+0000 7f25097fa640 4 set_mon_vals no callback set 2024-03-29T11:10:15.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.040+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2504003bf0 con 0x7f250c0a9d70 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.040+0000 7f24f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.040+0000 7f24f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.040+0000 7f24f37fe640 10 monclient: dump: 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.040+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f2504003e10 con 0x7f250c0a9d70 2024-03-29T11:10:15.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.041+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f2504084e70 con 0x7f250c0a9d70 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.041+0000 7f24f37fe640 10 client.4316.objecter ms_dispatch 0x7f250c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.041+0000 7f24f37fe640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.041+0000 7f24f37fe640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.041+0000 7f24f37fe640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.042+0000 7f250a7fc640 10 monclient: get_auth_request con 0x7f24f806dc90 auth_method 0 2024-03-29T11:10:15.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.043+0000 7f24f37fe640 10 client.4316.objecter ms_handle_connect 0x7f24f806dc90 2024-03-29T11:10:15.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.047+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25040507d0 con 0x7f250c0a9d70 2024-03-29T11:10:15.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.047+0000 7f24f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:15.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.047+0000 7f24f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.371+0000 7f24f17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.371+0000 7f24f17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.371+0000 7f24f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.371+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f250c06de90 con 0x7f250c0a9d70 2024-03-29T11:10:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.375+0000 7f24f37fe640 1 -- 172.21.15.105:0/1481234078 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f2504052700 con 0x7f250c0a9d70 2024-03-29T11:10:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.375+0000 7f24f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.375+0000 7f24f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:15.354 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:15.354 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:15.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.379+0000 7f24f17fa640 10 librados: watch_flush enter 2024-03-29T11:10:15.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 10 librados: watch_flush exit 2024-03-29T11:10:15.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:10:15.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:10:15.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 >> v1:172.21.15.110:6816/1087970004 conn(0x7f24f806dc90 legacy=0x7f24f8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:15.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 10 monclient: shutdown 2024-03-29T11:10:15.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:15.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.380+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 >> v1:172.21.15.105:6790/0 conn(0x7f250c0a9d70 legacy=0x7f250c1e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:15.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.382+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 shutdown_connections 2024-03-29T11:10:15.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.382+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 >> 172.21.15.105:0/1481234078 conn(0x7f250c066a00 msgr2=0x7f250c13dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:15.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.382+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 shutdown_connections 2024-03-29T11:10:15.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.382+0000 7f24f17fa640 1 -- 172.21.15.105:0/1481234078 wait complete. 2024-03-29T11:10:15.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:15.382+0000 7f24f17fa640 1 librados: shutdown 2024-03-29T11:10:15.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:22.186 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.368+0000 7fe7fe36d640 10 monclient: get_monmap_and_config 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.368+0000 7fe7fe36d640 10 monclient: build_initial_monmap 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.368+0000 7fe7fe36d640 10 monclient: monmap: 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:22.369485+0000 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:22.369485+0000 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:22.347 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:22.348 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:22.348 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.369+0000 7fe7fe36d640 1 Processor -- start 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.369+0000 7fe7fe36d640 1 -- start start 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.369+0000 7fe7fe36d640 10 monclient: init 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:22.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f80b0860 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f813ef30 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f80af8e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b0c40 con 0x7fe7f813ef30 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80aed10 con 0x7fe7f80b0860 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80afd50 con 0x7fe7f80af8e0 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:22.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.370+0000 7fe7fe36d640 10 monclient(hunting): authenticate will time out at 896.646532s 2024-03-29T11:10:22.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.375+0000 7fe7f7fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe7f80b0860 0x7fe7f813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:45557/0 (socket says 172.21.15.105:45557) 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.375+0000 7fe7f7fff640 1 -- 172.21.15.105:0/1432545613 learned_addr learned my addr 172.21.15.105:0/1432545613 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700915720 0 0) 0x7fe7f80aed10 con 0x7fe7f80b0860 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:10:22.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.376+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0003520 con 0x7fe7f80b0860 2024-03-29T11:10:22.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 800527006 0 0) 0x7fe7e0003520 con 0x7fe7f80b0860 2024-03-29T11:10:22.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 >> v1:172.21.15.105:6790/0 conn(0x7fe7f80af8e0 legacy=0x7fe7f8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 >> v1:172.21.15.105:6789/0 conn(0x7fe7f813ef30 legacy=0x7fe7f80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:22.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:22.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80ab1b0 con 0x7fe7f80b0860 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7fe36d640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7fe36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe7e8004a10 con 0x7fe7f80b0860 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: dump: 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:22.360 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7fe36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 1 -- 172.21.15.105:0/1432545613 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e8004ce0 con 0x7fe7f80b0860 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7fe36d640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.378+0000 7fe7fe36d640 4 set_mon_vals no callback set 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.379+0000 7fe7fe36d640 10 monclient: shutdown 2024-03-29T11:10:22.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.379+0000 7fe7fe36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:22.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.379+0000 7fe7fe36d640 1 -- 172.21.15.105:0/1432545613 >> v1:172.21.15.110:6789/0 conn(0x7fe7f80b0860 legacy=0x7fe7f813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.379+0000 7fe7fe36d640 1 -- 172.21.15.105:0/1432545613 shutdown_connections 2024-03-29T11:10:22.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.379+0000 7fe7fe36d640 1 -- 172.21.15.105:0/1432545613 >> 172.21.15.105:0/1432545613 conn(0x7fe7f806b660 msgr2=0x7fe7f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.381+0000 7fe7fe36d640 1 -- 172.21.15.105:0/1432545613 shutdown_connections 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.381+0000 7fe7fe36d640 1 -- 172.21.15.105:0/1432545613 wait complete. 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.382+0000 7fe7fe36d640 10 monclient: build_initial_monmap 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.382+0000 7fe7fe36d640 10 monclient: monmap: 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:22.363 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 librados: starting msgr at 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 librados: starting objecter 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 Processor -- start 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 -- start start 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 librados: setting wanted keys 2024-03-29T11:10:22.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 1 librados: calling monclient init 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.383+0000 7fe7fe36d640 10 monclient: init 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f80af8e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f813ef30 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f806ddb0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f806eac0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f806eda0 con 0x7fe7f80af8e0 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b3640 con 0x7fe7f813ef30 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:22.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.384+0000 7fe7fe36d640 10 monclient(hunting): authenticate will time out at 896.660217s 2024-03-29T11:10:22.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7fc90a640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fe7f806ddb0 0x7fe7f81e61d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:57403/0 (socket says 172.21.15.105:57403) 2024-03-29T11:10:22.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7fc90a640 1 -- 172.21.15.105:0/4158203626 learned_addr learned my addr 172.21.15.105:0/4158203626 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:22.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7f4ff9640 10 client.?.objecter ms_handle_connect 0x7fe7f813ef30 2024-03-29T11:10:22.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:22.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7f4ff9640 10 client.?.objecter ms_handle_connect 0x7fe7f806ddb0 2024-03-29T11:10:22.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:22.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.390+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961919554 0 0) 0x7fe7f806eac0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4004eb0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3794995142 0 0) 0x7fe7e4004eb0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7e4005300 con 0x7fe7f806ddb0 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe7f0001cb0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.371 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:22.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.391+0000 7fe7f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:22.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2836914669 0 0) 0x7fe7e4005300 con 0x7fe7f806ddb0 2024-03-29T11:10:22.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 >> v1:172.21.15.105:6790/0 conn(0x7fe7f813ef30 legacy=0x7fe7f80b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.373 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 >> v1:172.21.15.110:6789/0 conn(0x7fe7f80af8e0 legacy=0x7fe7f81def00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:10:22.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 1 monclient: found mon.a 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80b4150 con 0x7fe7f806ddb0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 10 monclient: _renew_subs 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f80b4420 con 0x7fe7f806ddb0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 10 monclient: _renew_subs 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:22.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f81e7ad0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.392+0000 7fe7fe36d640 1 librados: init done 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7fe36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7fe36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7fe36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f806da30 con 0x7fe7f806ddb0 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f0003970 con 0x7fe7f806ddb0 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.393+0000 7fe7f67fc640 4 set_mon_vals no callback set 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe7f0001cb0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 10 monclient: dump: 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:22.379 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:22.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fe7f0003be0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fe7f0084b70 con 0x7fe7f806ddb0 2024-03-29T11:10:22.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 10 client.4290.objecter ms_dispatch 0x7fe7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:22.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:22.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:22.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.398+0000 7fe7f4ff9640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:10:22.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.402+0000 7fe7f7fff640 10 monclient: get_auth_request con 0x7fe7e406e1d0 auth_method 0 2024-03-29T11:10:22.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.403+0000 7fe7f4ff9640 10 client.4290.objecter ms_handle_connect 0x7fe7e406e1d0 2024-03-29T11:10:22.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.403+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7f0050300 con 0x7fe7f806ddb0 2024-03-29T11:10:22.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.403+0000 7fe7f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:22.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.403+0000 7fe7f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.719+0000 7fe7fe36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.719+0000 7fe7fe36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.719+0000 7fe7fe36d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.719+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7f80b37d0 con 0x7fe7f806ddb0 2024-03-29T11:10:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.720+0000 7fe7f4ff9640 1 -- 172.21.15.105:0/4158203626 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fe7f0052400 con 0x7fe7f806ddb0 2024-03-29T11:10:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.720+0000 7fe7f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.720+0000 7fe7f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:22.699 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:22.699 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 10 librados: watch_flush enter 2024-03-29T11:10:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 10 librados: watch_flush exit 2024-03-29T11:10:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 >> v1:172.21.15.110:6816/1087970004 conn(0x7fe7e406e1d0 legacy=0x7fe7e4070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 10 monclient: shutdown 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 >> v1:172.21.15.105:6789/0 conn(0x7fe7f806ddb0 legacy=0x7fe7f81e61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:22.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 shutdown_connections 2024-03-29T11:10:22.704 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.724+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 >> 172.21.15.105:0/4158203626 conn(0x7fe7f806b660 msgr2=0x7fe7f80ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:22.704 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.725+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 shutdown_connections 2024-03-29T11:10:22.704 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.725+0000 7fe7fe36d640 1 -- 172.21.15.105:0/4158203626 wait complete. 2024-03-29T11:10:22.704 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:22.725+0000 7fe7fe36d640 1 librados: shutdown 2024-03-29T11:10:22.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:29.526 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:29.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.697+0000 7f8184c9d640 10 monclient: get_monmap_and_config 2024-03-29T11:10:29.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.697+0000 7f8184c9d640 10 monclient: build_initial_monmap 2024-03-29T11:10:29.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.697+0000 7f8184c9d640 10 monclient: monmap: 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:29.698235+0000 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:29.698235+0000 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:29.676 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.698+0000 7f8184c9d640 1 Processor -- start 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.698+0000 7f8184c9d640 1 -- start start 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.698+0000 7f8184c9d640 10 monclient: init 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.699+0000 7f8184c9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.699+0000 7f8184c9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:29.677 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.699+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f81800b4920 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:29.681 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.702+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f81800b4200 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:29.681 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.702+0000 7f817effd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81800b4200 0x7f81800b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:51317/0 (socket says 172.21.15.105:51317) 2024-03-29T11:10:29.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.702+0000 7f817effd640 1 -- 172.21.15.105:0/3595629531 learned_addr learned my addr 172.21.15.105:0/3595629531 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:29.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.703+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f818013f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:29.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.703+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800b0930 con 0x7f81800b4920 2024-03-29T11:10:29.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.703+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800af460 con 0x7f81800b4200 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800aeda0 con 0x7f818013f9a0 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f8184c9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f8184c9d640 10 monclient(hunting): authenticate will time out at 903.980017s 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093063017 0 0) 0x7f81800b0930 con 0x7f81800b4920 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:29.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:29.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:10:29.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8168003520 con 0x7f81800b4920 2024-03-29T11:10:29.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061003023 0 0) 0x7f8168003520 con 0x7f81800b4920 2024-03-29T11:10:29.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.704+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 >> v1:172.21.15.105:6790/0 conn(0x7f818013f9a0 legacy=0x7f818014d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:29.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 >> v1:172.21.15.110:6789/0 conn(0x7f81800b4200 legacy=0x7f81800b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:29.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:29.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:10:29.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:29.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81800af980 con 0x7f81800b4920 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8170001690 con 0x7f81800b4920 2024-03-29T11:10:29.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 10 monclient: dump: 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.692 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f8184c9d640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f8184c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.706+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8170001920 con 0x7f81800b4920 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f817e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f817e7fc640 1 -- 172.21.15.105:0/3595629531 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8170001d30 con 0x7f81800b4920 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f817e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:29.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f817e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f817e7fc640 10 monclient: dump: 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.695 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f8184c9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f8184c9d640 4 set_mon_vals no callback set 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f8184c9d640 10 monclient: shutdown 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f8184c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.707+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 >> v1:172.21.15.105:6789/0 conn(0x7f81800b4920 legacy=0x7f818013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.711+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 shutdown_connections 2024-03-29T11:10:29.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.711+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 >> 172.21.15.105:0/3595629531 conn(0x7f818006d230 msgr2=0x7f8180066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:29.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.711+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 shutdown_connections 2024-03-29T11:10:29.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.711+0000 7f8184c9d640 1 -- 172.21.15.105:0/3595629531 wait complete. 2024-03-29T11:10:29.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.712+0000 7f8184c9d640 10 monclient: build_initial_monmap 2024-03-29T11:10:29.703 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.712+0000 7f8184c9d640 10 monclient: monmap: 2024-03-29T11:10:29.703 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:29.703 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:29.703 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:29.704 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 librados: starting msgr at 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 librados: starting objecter 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 Processor -- start 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 -- start start 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 librados: setting wanted keys 2024-03-29T11:10:29.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 1 librados: calling monclient init 2024-03-29T11:10:29.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.713+0000 7f8184c9d640 10 monclient: init 2024-03-29T11:10:29.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.714+0000 7f8184c9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:29.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.714+0000 7f8184c9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:29.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.714+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f81800b4200 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:29.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.716+0000 7f817f7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f81800b4200 0x7f818013a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:56657/0 (socket says 172.21.15.105:56657) 2024-03-29T11:10:29.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.717+0000 7f817f7fe640 1 -- 172.21.15.105:0/1999632369 learned_addr learned my addr 172.21.15.105:0/1999632369 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:29.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.717+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f81800b4920 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:29.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f818013f9a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:29.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800ae3f0 con 0x7f81800b4200 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818006f0d0 con 0x7f818013f9a0 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818006f3b0 con 0x7f81800b4920 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8184c9d640 10 monclient(hunting): authenticate will time out at 903.994247s 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8167fff640 10 client.?.objecter ms_handle_connect 0x7f81800b4200 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8167fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250299144 0 0) 0x7f81800ae3f0 con 0x7f81800b4200 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8167fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.718+0000 7f8167fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.719+0000 7f8167fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.719+0000 7f8167fff640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.719+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c004eb0 con 0x7f81800b4200 2024-03-29T11:10:29.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320810521 0 0) 0x7f816c004eb0 con 0x7f81800b4200 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81800ae3f0 con 0x7f81800b4200 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8170001650 con 0x7f81800b4200 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 monclient(hunting): dump: 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:29.711 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 client.?.objecter ms_handle_connect 0x7f818013f9a0 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.720+0000 7f8167fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 696436400 0 0) 0x7f81800ae3f0 con 0x7f81800b4200 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 >> v1:172.21.15.105:6790/0 conn(0x7f81800b4920 legacy=0x7f81800abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 >> v1:172.21.15.110:6789/0 conn(0x7f818013f9a0 legacy=0x7f818006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 1 monclient: found mon.a 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818006de90 con 0x7f81800b4200 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 monclient: _finish_auth 0 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:29.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 client.?.objecter ms_handle_connect 0x7f81800b4920 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.721+0000 7f8167fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81700039e0 con 0x7f81800b4200 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8170003db0 con 0x7f81800b4200 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.722+0000 7f8167fff640 10 monclient: dump: 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:29.713 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:29.714 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:29.714 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:29.714 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:29.714 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.723+0000 7f817dffb640 4 set_mon_vals no callback set 2024-03-29T11:10:29.715 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 10 monclient: _renew_subs 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818006e0e0 con 0x7f81800b4200 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.724+0000 7f8184c9d640 10 monclient: _renew_subs 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.725+0000 7f8184c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.725+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f818013b270 con 0x7f81800b4200 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.725+0000 7f8184c9d640 1 librados: init done 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.726+0000 7f8184c9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.726+0000 7f8184c9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.726+0000 7f8184c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.726+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81800ae1e0 con 0x7f81800b4200 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.727+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8170003f10 con 0x7f81800b4200 2024-03-29T11:10:29.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f817effd640 10 monclient: get_auth_request con 0x7f816c06db80 auth_method 0 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8170084cf0 con 0x7f81800b4200 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 10 client.4296.objecter ms_dispatch 0x7f8180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.728+0000 7f8167fff640 10 client.4296.objecter ms_handle_connect 0x7f816c06db80 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.736+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81800ae1e0 con 0x7f81800b4200 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.736+0000 7f8167fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:29.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:29.736+0000 7f8167fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:30.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.060+0000 7f8184c9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:30.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.060+0000 7f8184c9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:30.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.060+0000 7f8184c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:10:30.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.060+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f818006e270 con 0x7f81800b4200 2024-03-29T11:10:30.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.062+0000 7f8167fff640 1 -- 172.21.15.105:0/1999632369 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f81700505d0 con 0x7f81800b4200 2024-03-29T11:10:30.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.062+0000 7f8167fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:30.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.062+0000 7f8167fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:30.041 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:30.042 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:30.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.066+0000 7f8184c9d640 10 librados: watch_flush enter 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 10 librados: watch_flush exit 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 >> v1:172.21.15.110:6816/1087970004 conn(0x7f816c06db80 legacy=0x7f816c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 10 monclient: shutdown 2024-03-29T11:10:30.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 >> v1:172.21.15.105:6789/0 conn(0x7f81800b4200 legacy=0x7f818013a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.068+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 shutdown_connections 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.069+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 >> 172.21.15.105:0/1999632369 conn(0x7f818006d230 msgr2=0x7f81800b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.069+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 shutdown_connections 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.069+0000 7f8184c9d640 1 -- 172.21.15.105:0/1999632369 wait complete. 2024-03-29T11:10:30.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:30.069+0000 7f8184c9d640 1 librados: shutdown 2024-03-29T11:10:30.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:36.872 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:37.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.045+0000 7f0acee79640 10 monclient: get_monmap_and_config 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.045+0000 7f0acee79640 10 monclient: build_initial_monmap 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.045+0000 7f0acee79640 10 monclient: monmap: 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:37.046220+0000 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:37.046220+0000 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:37.024 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.046+0000 7f0acee79640 1 Processor -- start 2024-03-29T11:10:37.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.046+0000 7f0acee79640 1 -- start start 2024-03-29T11:10:37.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.046+0000 7f0acee79640 10 monclient: init 2024-03-29T11:10:37.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:37.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:37.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac002b4c0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac000b8a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac002a5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac002c5c0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00c11a0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac002afa0 con 0x7f0ac002b4c0 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:37.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.047+0000 7f0acee79640 10 monclient(hunting): authenticate will time out at 911.323348s 2024-03-29T11:10:37.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.050+0000 7f0acde77640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f0ac002b4c0 0x7f0ac0006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:37599/0 (socket says 172.21.15.105:37599) 2024-03-29T11:10:37.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.050+0000 7f0acde77640 1 -- 172.21.15.105:0/463514113 learned_addr learned my addr 172.21.15.105:0/463514113 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:37.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981661652 0 0) 0x7f0ac002afa0 con 0x7f0ac002b4c0 2024-03-29T11:10:37.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab4003520 con 0x7f0ac002b4c0 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336539127 0 0) 0x7f0ac002c5c0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.056+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac002afa0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856299409 0 0) 0x7f0ac00c11a0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:10:37.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac002c5c0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232402054 0 0) 0x7f0ac002afa0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 >> v1:172.21.15.105:6790/0 conn(0x7f0ac002b4c0 legacy=0x7f0ac0006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 >> v1:172.21.15.110:6789/0 conn(0x7f0ac002a5a0 legacy=0x7f0ac00c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: _finish_hunting 0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 monclient: found mon.noname-a 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00cb160 con 0x7f0ac000b8a0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: _finish_auth 0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: _check_auth_tickets 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0ac804f730 con 0x7f0ac000b8a0 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:37.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: dump: 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acee79640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acee79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ac8075f10 con 0x7f0ac000b8a0 2024-03-29T11:10:37.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acee79640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acee79640 4 set_mon_vals no callback set 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 1 -- 172.21.15.105:0/463514113 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0ac80758b0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.057+0000 7f0acce75640 10 monclient: dump: 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:37.040 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 10 monclient: shutdown 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 1 -- 172.21.15.105:0/463514113 >> v1:172.21.15.105:6789/0 conn(0x7f0ac000b8a0 legacy=0x7f0ac00c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 1 -- 172.21.15.105:0/463514113 shutdown_connections 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 1 -- 172.21.15.105:0/463514113 >> 172.21.15.105:0/463514113 conn(0x7f0ac00058f0 msgr2=0x7f0ac0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:37.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 1 -- 172.21.15.105:0/463514113 shutdown_connections 2024-03-29T11:10:37.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.058+0000 7f0acee79640 1 -- 172.21.15.105:0/463514113 wait complete. 2024-03-29T11:10:37.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.059+0000 7f0acee79640 10 monclient: build_initial_monmap 2024-03-29T11:10:37.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.059+0000 7f0acee79640 10 monclient: monmap: 2024-03-29T11:10:37.043 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 librados: starting msgr at 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 librados: starting objecter 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 Processor -- start 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 -- start start 2024-03-29T11:10:37.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 librados: setting wanted keys 2024-03-29T11:10:37.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 1 librados: calling monclient init 2024-03-29T11:10:37.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.060+0000 7f0acee79640 10 monclient: init 2024-03-29T11:10:37.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:37.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:37.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac000b8a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac002a5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac002b4c0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00c1a30 con 0x7f0ac002b4c0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00c2b50 con 0x7f0ac000b8a0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00c2dc0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:37.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.061+0000 7f0acee79640 10 monclient(hunting): authenticate will time out at 911.337111s 2024-03-29T11:10:37.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.066+0000 7f0acde77640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0ac000b8a0 0x7f0ac00ca970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:60123/0 (socket says 172.21.15.105:60123) 2024-03-29T11:10:37.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.066+0000 7f0acde77640 1 -- 172.21.15.105:0/1873552467 learned_addr learned my addr 172.21.15.105:0/1873552467 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:37.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.066+0000 7f0abe7fc640 10 client.?.objecter ms_handle_connect 0x7f0ac000b8a0 2024-03-29T11:10:37.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:37.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 client.?.objecter ms_handle_connect 0x7f0ac002b4c0 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 client.?.objecter ms_handle_connect 0x7f0ac002a5a0 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473000393 0 0) 0x7f0ac00c2b50 con 0x7f0ac000b8a0 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aac004eb0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.067+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567922936 0 0) 0x7f0ac00c2dc0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac00c2b50 con 0x7f0ac002a5a0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835368179 0 0) 0x7f0ac00c1a30 con 0x7f0ac002b4c0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac00c2dc0 con 0x7f0ac002b4c0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321044104 0 0) 0x7f0aac004eb0 con 0x7f0ac000b8a0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ac00c1a30 con 0x7f0ac000b8a0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0ab8001950 con 0x7f0ac000b8a0 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:37.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): dump: 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:37.056 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.068+0000 7f0abe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377385012 0 0) 0x7f0ac00c2b50 con 0x7f0ac002a5a0 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0aac004eb0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0ac804f400 con 0x7f0ac002a5a0 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 10 monclient(hunting): dump: 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.057 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 12136412 0 0) 0x7f0aac004eb0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 >> v1:172.21.15.110:6789/0 conn(0x7f0ac000b8a0 legacy=0x7f0ac00ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.069+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 >> v1:172.21.15.105:6789/0 conn(0x7f0ac002b4c0 legacy=0x7f0ac016ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 1 monclient: found mon.c 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00c30a0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0abe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 10 monclient: _renew_subs 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac00c3380 con 0x7f0ac002a5a0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 10 monclient: _renew_subs 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00b6e30 con 0x7f0ac002a5a0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.070+0000 7f0acee79640 1 librados: init done 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.071+0000 7f0acee79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.071+0000 7f0acee79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.071+0000 7f0acee79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:37.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.071+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac00c17a0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.060 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ac804fdd0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.060 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:37.060 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0ac8067bf0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 10 monclient: dump: 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.073+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f0ac8067e10 con 0x7f0ac002a5a0 2024-03-29T11:10:37.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.074+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f0ac80fe1e0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.074+0000 7f0abe7fc640 10 client.4328.objecter ms_dispatch 0x7f0ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.074+0000 7f0abe7fc640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.074+0000 7f0abe7fc640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.074+0000 7f0abe7fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.077+0000 7f0abffff640 4 set_mon_vals no callback set 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.081+0000 7f0acde77640 10 monclient: get_auth_request con 0x7f0aac06dfc0 auth_method 0 2024-03-29T11:10:37.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.082+0000 7f0abe7fc640 10 client.4328.objecter ms_handle_connect 0x7f0aac06dfc0 2024-03-29T11:10:37.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.087+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ac80c9b40 con 0x7f0ac002a5a0 2024-03-29T11:10:37.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.087+0000 7f0abe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:37.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.087+0000 7f0abe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:37.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.442+0000 7f0acee79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:37.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.442+0000 7f0acee79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:37.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.442+0000 7f0acee79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:37.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.442+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ac0004cd0 con 0x7f0ac002a5a0 2024-03-29T11:10:37.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.446+0000 7f0abe7fc640 1 -- 172.21.15.105:0/1873552467 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f0ac80cba70 con 0x7f0ac002a5a0 2024-03-29T11:10:37.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.446+0000 7f0abe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:37.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.446+0000 7f0abe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:37.424 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:37.424 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:37.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.448+0000 7f0acee79640 10 librados: watch_flush enter 2024-03-29T11:10:37.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.448+0000 7f0acee79640 10 librados: watch_flush exit 2024-03-29T11:10:37.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.448+0000 7f0acee79640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.448+0000 7f0acee79640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 >> v1:172.21.15.110:6816/1087970004 conn(0x7f0aac06dfc0 legacy=0x7f0aac070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 10 monclient: shutdown 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 >> v1:172.21.15.105:6790/0 conn(0x7f0ac002a5a0 legacy=0x7f0ac00c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 shutdown_connections 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 >> 172.21.15.105:0/1873552467 conn(0x7f0ac00058f0 msgr2=0x7f0ac00aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 shutdown_connections 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 -- 172.21.15.105:0/1873552467 wait complete. 2024-03-29T11:10:37.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:37.449+0000 7f0acee79640 1 librados: shutdown 2024-03-29T11:10:37.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:44.248 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.424+0000 7f58f051c640 10 monclient: get_monmap_and_config 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.424+0000 7f58f051c640 10 monclient: build_initial_monmap 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.424+0000 7f58f051c640 10 monclient: monmap: 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:44.425872+0000 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:44.425872+0000 2024-03-29T11:10:44.403 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.425+0000 7f58f051c640 1 Processor -- start 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.425+0000 7f58f051c640 1 -- start start 2024-03-29T11:10:44.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.425+0000 7f58f051c640 10 monclient: init 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-a con 0x7f58e806d9d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-b con 0x7f58e80af1d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-c con 0x7f58e8147a60 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e80aed10 con 0x7f58e806d9d0 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e80afd10 con 0x7f58e8147a60 2024-03-29T11:10:44.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e80ab490 con 0x7f58e80af1d0 2024-03-29T11:10:44.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:44.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.426+0000 7f58f051c640 10 monclient(hunting): authenticate will time out at 918.702919s 2024-03-29T11:10:44.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.431+0000 7f58edab7640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f58e80af1d0 0x7f58e80af600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:43940/0 (socket says 172.21.15.105:43940) 2024-03-29T11:10:44.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.431+0000 7f58edab7640 1 -- 172.21.15.105:0/6916848 learned_addr learned my addr 172.21.15.105:0/6916848 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:44.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189591662 0 0) 0x7f58e80ab490 con 0x7f58e80af1d0 2024-03-29T11:10:44.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 10 monclient(hunting): _negotiate 2024-03-29T11:10:44.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:44.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:44.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:10:44.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.432+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d4003520 con 0x7f58e80af1d0 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052664286 0 0) 0x7f58e80aed10 con 0x7f58e806d9d0 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient(hunting): _negotiate 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:10:44.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e80ab490 con 0x7f58e806d9d0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498906330 0 0) 0x7f58d4003520 con 0x7f58e80af1d0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 >> v1:172.21.15.110:6789/0 conn(0x7f58e8147a60 legacy=0x7f58e8149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 >> v1:172.21.15.105:6789/0 conn(0x7f58e806d9d0 legacy=0x7f58e80b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: _finish_hunting 0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 monclient: found mon.noname-b 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80ab770 con 0x7f58e80af1d0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: _finish_auth 0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: _check_auth_tickets 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f58d8001690 con 0x7f58e80af1d0 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:44.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: dump: 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58f051c640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58f051c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:44.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d80017f0 con 0x7f58e80af1d0 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.433+0000 7f58ed2b6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58ed2b6640 1 -- 172.21.15.105:0/6916848 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f58d8001c00 con 0x7f58e80af1d0 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58ed2b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58ed2b6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58ed2b6640 10 monclient: dump: 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:44.415 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:44.416 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58f051c640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:44.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.434+0000 7f58f051c640 4 set_mon_vals no callback set 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 10 monclient: shutdown 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 1 -- 172.21.15.105:0/6916848 >> v1:172.21.15.105:6790/0 conn(0x7f58e80af1d0 legacy=0x7f58e80af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 1 -- 172.21.15.105:0/6916848 shutdown_connections 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 1 -- 172.21.15.105:0/6916848 >> 172.21.15.105:0/6916848 conn(0x7f58e806b660 msgr2=0x7f58e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:44.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 1 -- 172.21.15.105:0/6916848 shutdown_connections 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.435+0000 7f58f051c640 1 -- 172.21.15.105:0/6916848 wait complete. 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 10 monclient: build_initial_monmap 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 10 monclient: monmap: 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 librados: starting msgr at 2024-03-29T11:10:44.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 librados: starting objecter 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 Processor -- start 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 -- start start 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 librados: setting wanted keys 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 1 librados: calling monclient init 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.436+0000 7f58f051c640 10 monclient: init 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.437+0000 7f58f051c640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:44.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.437+0000 7f58f051c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:44.421 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-a con 0x7f58e806d9d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-c con 0x7f58e80af1d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58ee2b8640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f58e806d9d0 0x7f58e80b3600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:39817/0 (socket says 172.21.15.105:39817) 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58ee2b8640 1 -- 172.21.15.105:0/1821053098 learned_addr learned my addr 172.21.15.105:0/1821053098 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58deffd640 10 client.?.objecter ms_handle_connect 0x7f58e806d9d0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58deffd640 10 client.?.objecter ms_handle_connect 0x7f58e80af1d0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.440+0000 7f58deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 10 monclient(hunting): picked mon.noname-b con 0x7f58e8147a60 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e81d4100 con 0x7f58e806d9d0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e813a5d0 con 0x7f58e8147a60 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e813a760 con 0x7f58e80af1d0 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:44.422 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58f051c640 10 monclient(hunting): authenticate will time out at 918.717236s 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467000722 0 0) 0x7f58e81d4100 con 0x7f58e806d9d0 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc004eb0 con 0x7f58e806d9d0 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489267684 0 0) 0x7f58e80ab770 con 0x7f58e80af1d0 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.441+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e81d4100 con 0x7f58e80af1d0 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309971487 0 0) 0x7f58e81d4100 con 0x7f58e80af1d0 2024-03-29T11:10:44.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e80ab770 con 0x7f58e80af1d0 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f58d8001960 con 0x7f58e80af1d0 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): dump: 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350494755 0 0) 0x7f58cc004eb0 con 0x7f58e806d9d0 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e81d4100 con 0x7f58e806d9d0 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f58e0001da0 con 0x7f58e806d9d0 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.442+0000 7f58deffd640 10 monclient(hunting): dump: 2024-03-29T11:10:44.425 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:44.426 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.443+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 29715311 0 0) 0x7f58e80ab770 con 0x7f58e80af1d0 2024-03-29T11:10:44.427 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.443+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 >> v1:172.21.15.110:6789/0 conn(0x7f58e8147a60 legacy=0x7f58e81d3900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 >> v1:172.21.15.105:6789/0 conn(0x7f58e806d9d0 legacy=0x7f58e80b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 1 monclient: found mon.c 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80b16b0 con 0x7f58e80af1d0 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58f051c640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58f051c640 10 monclient: _renew_subs 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58f051c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.444+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e80b11d0 con 0x7f58e80af1d0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.445+0000 7f58f051c640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.445+0000 7f58f051c640 10 monclient: _renew_subs 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.445+0000 7f58f051c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.445+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e80b3ef0 con 0x7f58e80af1d0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.445+0000 7f58f051c640 1 librados: init done 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.446+0000 7f58f051c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.446+0000 7f58f051c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.446+0000 7f58f051c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.446+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d80047e0 con 0x7f58e80af1d0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.446+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e8075f10 con 0x7f58e80af1d0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f58d8004af0 con 0x7f58e80af1d0 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:44.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 10 monclient: dump: 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f58d801aac0 con 0x7f58e80af1d0 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.447+0000 7f58ecab5640 4 set_mon_vals no callback set 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.448+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f58d80849a0 con 0x7f58e80af1d0 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.448+0000 7f58deffd640 10 client.4334.objecter ms_dispatch 0x7f58e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:44.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.448+0000 7f58deffd640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:44.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.448+0000 7f58deffd640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:44.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.448+0000 7f58deffd640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:10:44.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.449+0000 7f58ee2b8640 10 monclient: get_auth_request con 0x7f58cc06de40 auth_method 0 2024-03-29T11:10:44.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.449+0000 7f58deffd640 10 client.4334.objecter ms_handle_connect 0x7f58cc06de40 2024-03-29T11:10:44.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.456+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58d8050300 con 0x7f58e80af1d0 2024-03-29T11:10:44.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.456+0000 7f58deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:44.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.456+0000 7f58deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:44.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.849+0000 7f58f051c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:44.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.849+0000 7f58f051c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:44.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.849+0000 7f58f051c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:44.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.849+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e80b4080 con 0x7f58e80af1d0 2024-03-29T11:10:44.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.850+0000 7f58deffd640 1 -- 172.21.15.105:0/1821053098 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f58e80b4080 con 0x7f58e80af1d0 2024-03-29T11:10:44.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.850+0000 7f58deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:44.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.850+0000 7f58deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:44.828 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:44.829 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:44.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 10 librados: watch_flush enter 2024-03-29T11:10:44.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 10 librados: watch_flush exit 2024-03-29T11:10:44.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:10:44.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 >> v1:172.21.15.110:6816/1087970004 conn(0x7f58cc06de40 legacy=0x7f58cc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 10 monclient: shutdown 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 >> v1:172.21.15.105:6790/0 conn(0x7f58e80af1d0 legacy=0x7f58e81d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 shutdown_connections 2024-03-29T11:10:44.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.854+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 >> 172.21.15.105:0/1821053098 conn(0x7f58e806b660 msgr2=0x7f58e80af600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:44.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.857+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 shutdown_connections 2024-03-29T11:10:44.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.857+0000 7f58f051c640 1 -- 172.21.15.105:0/1821053098 wait complete. 2024-03-29T11:10:44.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:44.857+0000 7f58f051c640 1 librados: shutdown 2024-03-29T11:10:44.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:51.658 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:51.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.822+0000 7f3d6b89c640 10 monclient: get_monmap_and_config 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.822+0000 7f3d6b89c640 10 monclient: build_initial_monmap 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.822+0000 7f3d6b89c640 10 monclient: monmap: 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:51.823543+0000 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:51.823543+0000 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:51.801 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.823+0000 7f3d6b89c640 1 Processor -- start 2024-03-29T11:10:51.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.823+0000 7f3d6b89c640 1 -- start start 2024-03-29T11:10:51.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.823+0000 7f3d6b89c640 10 monclient: init 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.824+0000 7f3d6b89c640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.824+0000 7f3d6b89c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.824+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d641422b0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d64142690 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d69638640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3d641422b0 0x7f3d6413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:47370/0 (socket says 172.21.15.105:47370) 2024-03-29T11:10:51.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d69e39640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f3d640b17d0 0x7f3d640b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:56868/0 (socket says 172.21.15.105:56868) 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d69638640 1 -- 172.21.15.105:0/2127199055 learned_addr learned my addr 172.21.15.105:0/2127199055 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d640b17d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d64142c80 con 0x7f3d64142690 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d640aae40 con 0x7f3d641422b0 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6413fbd0 con 0x7f3d640b17d0 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d6b89c640 10 monclient(hunting): authenticate will time out at 926.101425s 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290701931 0 0) 0x7f3d6413fbd0 con 0x7f3d640b17d0 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d50003520 con 0x7f3d640b17d0 2024-03-29T11:10:51.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388145041 0 0) 0x7f3d640aae40 con 0x7f3d641422b0 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6413fbd0 con 0x7f3d641422b0 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454103131 0 0) 0x7f3d64142c80 con 0x7f3d64142690 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.825+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d640aae40 con 0x7f3d64142690 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079274334 0 0) 0x7f3d50003520 con 0x7f3d640b17d0 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 >> v1:172.21.15.110:6789/0 conn(0x7f3d641422b0 legacy=0x7f3d6413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 >> v1:172.21.15.105:6789/0 conn(0x7f3d64142690 legacy=0x7f3d640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:51.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 monclient: found mon.noname-b 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64151940 con 0x7f3d640b17d0 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d5c001a10 con 0x7f3d640b17d0 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: dump: 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.807 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d6b89c640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d6b89c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d5c001b70 con 0x7f3d640b17d0 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 1 -- 172.21.15.105:0/2127199055 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d5c001f00 con 0x7f3d640b17d0 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d5bfff640 10 monclient: dump: 2024-03-29T11:10:51.808 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d6b89c640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:51.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.826+0000 7f3d6b89c640 4 set_mon_vals no callback set 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 10 monclient: shutdown 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 >> v1:172.21.15.105:6790/0 conn(0x7f3d640b17d0 legacy=0x7f3d640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 shutdown_connections 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 >> 172.21.15.105:0/2127199055 conn(0x7f3d64066a00 msgr2=0x7f3d64066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 shutdown_connections 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.827+0000 7f3d6b89c640 1 -- 172.21.15.105:0/2127199055 wait complete. 2024-03-29T11:10:51.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.828+0000 7f3d6b89c640 10 monclient: build_initial_monmap 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.828+0000 7f3d6b89c640 10 monclient: monmap: 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 librados: starting msgr at 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 librados: starting objecter 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 Processor -- start 2024-03-29T11:10:51.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 -- start start 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 librados: setting wanted keys 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 1 librados: calling monclient init 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.829+0000 7f3d6b89c640 10 monclient: init 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d640b17d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d641422b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d64142690 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d641d7660 con 0x7f3d641422b0 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d640b1400 con 0x7f3d64142690 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d69638640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f3d640b17d0 0x7f3d641d6a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:57124/0 (socket says 172.21.15.105:57124) 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d69638640 1 -- 172.21.15.105:0/3753309000 learned_addr learned my addr 172.21.15.105:0/3753309000 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d641d7160 con 0x7f3d640b17d0 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:51.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d6b89c640 10 monclient(hunting): authenticate will time out at 926.106762s 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d59ffb640 10 client.?.objecter ms_handle_connect 0x7f3d640b17d0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.830+0000 7f3d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 client.?.objecter ms_handle_connect 0x7f3d641422b0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127506766 0 0) 0x7f3d641d7160 con 0x7f3d640b17d0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48004eb0 con 0x7f3d640b17d0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 client.?.objecter ms_handle_connect 0x7f3d64142690 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344222281 0 0) 0x7f3d48004eb0 con 0x7f3d640b17d0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d641d7160 con 0x7f3d640b17d0 2024-03-29T11:10:51.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d600019f0 con 0x7f3d640b17d0 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): dump: 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:51.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313232200 0 0) 0x7f3d641d7660 con 0x7f3d641422b0 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48004eb0 con 0x7f3d641422b0 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389792715 0 0) 0x7f3d640b1400 con 0x7f3d64142690 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.831+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d641d7660 con 0x7f3d64142690 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2704600408 0 0) 0x7f3d641d7160 con 0x7f3d640b17d0 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 >> v1:172.21.15.110:6789/0 conn(0x7f3d64142690 legacy=0x7f3d641e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 >> v1:172.21.15.105:6789/0 conn(0x7f3d641422b0 legacy=0x7f3d641db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:51.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 monclient: found mon.c 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d641d72f0 con 0x7f3d640b17d0 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: _finish_auth 0 2024-03-29T11:10:51.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 10 monclient: _renew_subs 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6414e250 con 0x7f3d640b17d0 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d60001c90 con 0x7f3d640b17d0 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d60004c30 con 0x7f3d640b17d0 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:51.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d59ffb640 10 monclient: dump: 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 10 monclient: _renew_subs 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d5b7fe640 4 set_mon_vals no callback set 2024-03-29T11:10:51.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d641db930 con 0x7f3d640b17d0 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.832+0000 7f3d6b89c640 1 librados: init done 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.833+0000 7f3d6b89c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.834+0000 7f3d6b89c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.834+0000 7f3d6b89c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.834+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d64075f10 con 0x7f3d640b17d0 2024-03-29T11:10:51.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f3d6001ac00 con 0x7f3d640b17d0 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f3d60084c10 con 0x7f3d640b17d0 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 10 client.4340.objecter ms_dispatch 0x7f3d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.835+0000 7f3d59ffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.836+0000 7f3d68e37640 10 monclient: get_auth_request con 0x7f3d4806de80 auth_method 0 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.836+0000 7f3d59ffb640 10 client.4340.objecter ms_handle_connect 0x7f3d4806de80 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.843+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d60050530 con 0x7f3d640b17d0 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.843+0000 7f3d59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:51.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:51.843+0000 7f3d59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:52.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.221+0000 7f3d6b89c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:52.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.221+0000 7f3d6b89c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:52.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.221+0000 7f3d6b89c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:52.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.221+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d640b4ca0 con 0x7f3d640b17d0 2024-03-29T11:10:52.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.222+0000 7f3d59ffb640 1 -- 172.21.15.105:0/3753309000 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f3d60052460 con 0x7f3d640b17d0 2024-03-29T11:10:52.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.222+0000 7f3d59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:52.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.222+0000 7f3d59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:52.200 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:52.200 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.225+0000 7f3d6b89c640 10 librados: watch_flush enter 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.225+0000 7f3d6b89c640 10 librados: watch_flush exit 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.225+0000 7f3d6b89c640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.225+0000 7f3d6b89c640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 >> v1:172.21.15.110:6816/1087970004 conn(0x7f3d4806de80 legacy=0x7f3d48070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 10 monclient: shutdown 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:52.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 >> v1:172.21.15.105:6790/0 conn(0x7f3d640b17d0 legacy=0x7f3d641d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:52.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 shutdown_connections 2024-03-29T11:10:52.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.226+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 >> 172.21.15.105:0/3753309000 conn(0x7f3d64066a00 msgr2=0x7f3d6413db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:52.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.227+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 shutdown_connections 2024-03-29T11:10:52.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.227+0000 7f3d6b89c640 1 -- 172.21.15.105:0/3753309000 wait complete. 2024-03-29T11:10:52.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:52.227+0000 7f3d6b89c640 1 librados: shutdown 2024-03-29T11:10:52.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:59.027 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:59.158 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.179+0000 7f674dc40640 10 monclient: get_monmap_and_config 2024-03-29T11:10:59.158 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.179+0000 7f674dc40640 10 monclient: build_initial_monmap 2024-03-29T11:10:59.158 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.180+0000 7f674dc40640 10 monclient: monmap: 2024-03-29T11:10:59.158 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:59.158 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:10:59.181033+0000 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:10:59.181033+0000 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:10:59.159 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.180+0000 7f674dc40640 1 Processor -- start 2024-03-29T11:10:59.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.181+0000 7f674dc40640 1 -- start start 2024-03-29T11:10:59.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.181+0000 7f674dc40640 10 monclient: init 2024-03-29T11:10:59.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.182+0000 7f674dc40640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:59.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.182+0000 7f674dc40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:59.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-a con 0x7f67481457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:59.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-c con 0x7f674813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:59.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f6747fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f67480a9d70 0x7f674814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:39906/0 (socket says 172.21.15.105:39906) 2024-03-29T11:10:59.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f6747fff640 1 -- 172.21.15.105:0/1385078094 learned_addr learned my addr 172.21.15.105:0/1385078094 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-b con 0x7f67480a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480aaef0 con 0x7f67481457f0 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6748145bd0 con 0x7f674813f650 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6748145eb0 con 0x7f67480a9d70 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.183+0000 7f674dc40640 10 monclient(hunting): authenticate will time out at 933.459671s 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639099853 0 0) 0x7f6748145eb0 con 0x7f67480a9d70 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6730003520 con 0x7f67480a9d70 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4141484689 0 0) 0x7f6730003520 con 0x7f67480a9d70 2024-03-29T11:10:59.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 >> v1:172.21.15.110:6789/0 conn(0x7f674813f650 legacy=0x7f67480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.184+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 >> v1:172.21.15.105:6789/0 conn(0x7f67481457f0 legacy=0x7f674813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 1 monclient: found mon.noname-b 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748146340 con 0x7f67480a9d70 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6740001a10 con 0x7f67480a9d70 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: dump: 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:59.165 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f674dc40640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f674dc40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6740001be0 con 0x7f67480a9d70 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 1 -- 172.21.15.105:0/1385078094 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6740002030 con 0x7f67480a9d70 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:59.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f67467fc640 10 monclient: dump: 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f674dc40640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:59.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.185+0000 7f674dc40640 4 set_mon_vals no callback set 2024-03-29T11:10:59.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.186+0000 7f674dc40640 10 monclient: shutdown 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.186+0000 7f674dc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.186+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 >> v1:172.21.15.105:6790/0 conn(0x7f67480a9d70 legacy=0x7f674814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.186+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 shutdown_connections 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.186+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 >> 172.21.15.105:0/1385078094 conn(0x7f6748066a00 msgr2=0x7f6748066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.189+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 shutdown_connections 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.189+0000 7f674dc40640 1 -- 172.21.15.105:0/1385078094 wait complete. 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.190+0000 7f674dc40640 10 monclient: build_initial_monmap 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.190+0000 7f674dc40640 10 monclient: monmap: 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.169 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:10:59.170 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:10:59.170 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:10:59.170 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 librados: starting msgr at 2024-03-29T11:10:59.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 librados: starting objecter 2024-03-29T11:10:59.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 Processor -- start 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 -- start start 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 librados: setting wanted keys 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 1 librados: calling monclient init 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.191+0000 7f674dc40640 10 monclient: init 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.192+0000 7f674dc40640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-a con 0x7f67480a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-c con 0x7f674813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient(hunting): picked mon.noname-b con 0x7f67481457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f6746ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f674813f650 0x7f67481399a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41258/0 (socket says 172.21.15.105:41258) 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f6746ffd640 1 -- 172.21.15.105:0/4088117394 learned_addr learned my addr 172.21.15.105:0/4088117394 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f672ffff640 10 client.?.objecter ms_handle_connect 0x7f674813f650 2024-03-29T11:10:59.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f672ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480af3e0 con 0x7f67480a9d70 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f672ffff640 10 client.?.objecter ms_handle_connect 0x7f67481457f0 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f672ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480adcb0 con 0x7f67481457f0 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480ae1d0 con 0x7f674813f650 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.193+0000 7f674dc40640 10 monclient(hunting): authenticate will time out at 933.469940s 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693167465 0 0) 0x7f67480ae1d0 con 0x7f674813f650 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734004eb0 con 0x7f674813f650 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901108729 0 0) 0x7f67480adcb0 con 0x7f67481457f0 2024-03-29T11:10:59.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.194+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67480ae1d0 con 0x7f67481457f0 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655981760 0 0) 0x7f6734004eb0 con 0x7f674813f650 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67480adcb0 con 0x7f674813f650 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f673c001a30 con 0x7f674813f650 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 10 monclient(hunting): dump: 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:59.176 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1762134519 0 0) 0x7f67480adcb0 con 0x7f674813f650 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 >> v1:172.21.15.110:6789/0 conn(0x7f67481457f0 legacy=0x7f674813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:59.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.195+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 >> v1:172.21.15.105:6789/0 conn(0x7f67480a9d70 legacy=0x7f67481da6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 1 monclient: found mon.c 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480ae360 con 0x7f674813f650 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 10 monclient: _finish_auth 0 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f672ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f674dc40640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f674dc40640 10 monclient: _renew_subs 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f674dc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.196+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67480ad790 con 0x7f674813f650 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f673c001cf0 con 0x7f674813f650 2024-03-29T11:10:59.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f673c003a60 con 0x7f674813f650 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f672ffff640 10 monclient: dump: 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:10:59.182 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.197+0000 7f6745ffb640 4 set_mon_vals no callback set 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f674dc40640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f674dc40640 10 monclient: _renew_subs 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f674dc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67481dafb0 con 0x7f674813f650 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f674dc40640 1 librados: init done 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.199+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f673c003bc0 con 0x7f674813f650 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f67477fe640 10 monclient: get_auth_request con 0x7f673406dcb0 auth_method 0 2024-03-29T11:10:59.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 10 client.4346.objecter ms_handle_connect 0x7f673406dcb0 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f673c054a40 con 0x7f674813f650 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 10 client.4346.objecter ms_dispatch 0x7f6748000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.200+0000 7f672ffff640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.201+0000 7f674dc40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.201+0000 7f674dc40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:59.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.201+0000 7f674dc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.201+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6748075f10 con 0x7f674813f650 2024-03-29T11:10:59.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.209+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f673c050ce0 con 0x7f674813f650 2024-03-29T11:10:59.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.209+0000 7f672ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:59.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.209+0000 7f672ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:59.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.516+0000 7f674dc40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:59.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.516+0000 7f674dc40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:59.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.516+0000 7f674dc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:10:59.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.516+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67480b3680 con 0x7f674813f650 2024-03-29T11:10:59.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.517+0000 7f672ffff640 1 -- 172.21.15.105:0/4088117394 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f673c050e80 con 0x7f674813f650 2024-03-29T11:10:59.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.517+0000 7f672ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:59.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.517+0000 7f672ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:59.495 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:10:59.496 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:59.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 10 librados: watch_flush enter 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 10 librados: watch_flush exit 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 >> v1:172.21.15.110:6816/1087970004 conn(0x7f673406dcb0 legacy=0x7f6734070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 10 monclient: shutdown 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 >> v1:172.21.15.105:6790/0 conn(0x7f674813f650 legacy=0x7f67481399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 shutdown_connections 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 >> 172.21.15.105:0/4088117394 conn(0x7f6748066a00 msgr2=0x7f6748066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 shutdown_connections 2024-03-29T11:10:59.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.520+0000 7f674dc40640 1 -- 172.21.15.105:0/4088117394 wait complete. 2024-03-29T11:10:59.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:10:59.521+0000 7f674dc40640 1 librados: shutdown 2024-03-29T11:10:59.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:06.320 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:06.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.493+0000 7f810cf5a640 10 monclient: get_monmap_and_config 2024-03-29T11:11:06.472 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.493+0000 7f810cf5a640 10 monclient: build_initial_monmap 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.493+0000 7f810cf5a640 10 monclient: monmap: 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:06.494248+0000 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:06.494248+0000 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:06.473 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.494+0000 7f810cf5a640 1 Processor -- start 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.494+0000 7f810cf5a640 1 -- start start 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.494+0000 7f810cf5a640 10 monclient: init 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-b con 0x7f81080a9750 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:06.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-c con 0x7f81080adca0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-a con 0x7f8108146a80 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080abdc0 con 0x7f8108146a80 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080b0070 con 0x7f81080adca0 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080af880 con 0x7f81080a9750 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.495+0000 7f810cf5a640 10 monclient(hunting): authenticate will time out at 940.771301s 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.496+0000 7f81077fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f81080a9750 0x7f8108143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:36458/0 (socket says 172.21.15.105:36458) 2024-03-29T11:11:06.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.496+0000 7f81077fe640 1 -- 172.21.15.105:0/712359436 learned_addr learned my addr 172.21.15.105:0/712359436 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070889733 0 0) 0x7f81080af880 con 0x7f81080a9750 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:11:06.476 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.498+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0003520 con 0x7f81080a9750 2024-03-29T11:11:06.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973690234 0 0) 0x7f80f0003520 con 0x7f81080a9750 2024-03-29T11:11:06.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 >> v1:172.21.15.110:6789/0 conn(0x7f81080adca0 legacy=0x7f8108146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:06.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 >> v1:172.21.15.105:6789/0 conn(0x7f8108146a80 legacy=0x7f8108151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:06.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:06.477 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 monclient: found mon.noname-b 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81080ac030 con 0x7f81080a9750 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f810cf5a640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f810cf5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80f8001690 con 0x7f81080a9750 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:06.478 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: dump: 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:06.479 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f810cf5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:06.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 1 -- 172.21.15.105:0/712359436 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f8001920 con 0x7f81080a9750 2024-03-29T11:11:06.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f81067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:06.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f810cf5a640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:06.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.499+0000 7f810cf5a640 4 set_mon_vals no callback set 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.500+0000 7f810cf5a640 10 monclient: shutdown 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.500+0000 7f810cf5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.500+0000 7f810cf5a640 1 -- 172.21.15.105:0/712359436 >> v1:172.21.15.105:6790/0 conn(0x7f81080a9750 legacy=0x7f8108143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.501+0000 7f810cf5a640 1 -- 172.21.15.105:0/712359436 shutdown_connections 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.501+0000 7f810cf5a640 1 -- 172.21.15.105:0/712359436 >> 172.21.15.105:0/712359436 conn(0x7f81080673e0 msgr2=0x7f81080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.501+0000 7f810cf5a640 1 -- 172.21.15.105:0/712359436 shutdown_connections 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.501+0000 7f810cf5a640 1 -- 172.21.15.105:0/712359436 wait complete. 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.502+0000 7f810cf5a640 10 monclient: build_initial_monmap 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.502+0000 7f810cf5a640 10 monclient: monmap: 2024-03-29T11:11:06.481 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.502+0000 7f810cf5a640 1 librados: starting msgr at 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.502+0000 7f810cf5a640 1 librados: starting objecter 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.503+0000 7f810cf5a640 1 Processor -- start 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.503+0000 7f810cf5a640 1 -- start start 2024-03-29T11:11:06.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.503+0000 7f810cf5a640 1 librados: setting wanted keys 2024-03-29T11:11:06.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.503+0000 7f810cf5a640 1 librados: calling monclient init 2024-03-29T11:11:06.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.503+0000 7f810cf5a640 10 monclient: init 2024-03-29T11:11:06.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.504+0000 7f810cf5a640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:06.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.504+0000 7f810cf5a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:06.485 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.507+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-a con 0x7f81080a9750 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.507+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-b con 0x7f81080adca0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 10 monclient(hunting): picked mon.noname-c con 0x7f8108146a80 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108123c10 con 0x7f81080a9750 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108123ef0 con 0x7f81080adca0 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108124b90 con 0x7f8108146a80 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f810cf5a640 10 monclient(hunting): authenticate will time out at 940.784051s 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f8107fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f8108146a80 0x7f810813e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:45810/0 (socket says 172.21.15.105:45810) 2024-03-29T11:11:06.487 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.508+0000 7f8107fff640 1 -- 172.21.15.105:0/1339643731 learned_addr learned my addr 172.21.15.105:0/1339643731 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 client.?.objecter ms_handle_connect 0x7f8108146a80 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 client.?.objecter ms_handle_connect 0x7f81080a9750 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710064529 0 0) 0x7f8108124b90 con 0x7f8108146a80 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:06.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f4004eb0 con 0x7f8108146a80 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966299098 0 0) 0x7f8108123c10 con 0x7f81080a9750 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8108124b90 con 0x7f81080a9750 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123038622 0 0) 0x7f8108124b90 con 0x7f81080a9750 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.511+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8108123c10 con 0x7f81080a9750 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80f8004ae0 con 0x7f81080a9750 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): dump: 2024-03-29T11:11:06.492 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1109327465 0 0) 0x7f80f4004eb0 con 0x7f8108146a80 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8108124b90 con 0x7f8108146a80 2024-03-29T11:11:06.493 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80fc0017f0 con 0x7f8108146a80 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient(hunting): dump: 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.494 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3395717111 0 0) 0x7f8108123c10 con 0x7f81080a9750 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 >> v1:172.21.15.105:6790/0 conn(0x7f8108146a80 legacy=0x7f810813e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 >> v1:172.21.15.110:6789/0 conn(0x7f81080adca0 legacy=0x7f810806fb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 monclient: found mon.a 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108124d20 con 0x7f81080a9750 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient: _finish_auth 0 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f80effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f810cf5a640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f810cf5a640 10 monclient: _renew_subs 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f810cf5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:06.495 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.512+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81080725a0 con 0x7f81080a9750 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f810cf5a640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f810cf5a640 10 monclient: _renew_subs 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f810cf5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108070410 con 0x7f81080a9750 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f810cf5a640 1 librados: init done 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f8004ef0 con 0x7f81080a9750 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:06.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f81057fa640 4 set_mon_vals no callback set 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f80f8005200 con 0x7f81080a9750 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.513+0000 7f80effff640 10 monclient: dump: 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.498 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.514+0000 7f810cf5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.514+0000 7f810cf5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.514+0000 7f810cf5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.514+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8108072320 con 0x7f81080a9750 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.514+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f80f8001b10 con 0x7f81080a9750 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.515+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f80f8084bf0 con 0x7f81080a9750 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.515+0000 7f80effff640 10 client.4320.objecter ms_dispatch 0x7f8108000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:06.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.515+0000 7f80effff640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:06.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.515+0000 7f80effff640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:06.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.515+0000 7f80effff640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T11:11:06.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.518+0000 7f8106ffd640 10 monclient: get_auth_request con 0x7f80f406dfe0 auth_method 0 2024-03-29T11:11:06.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.519+0000 7f80effff640 10 client.4320.objecter ms_handle_connect 0x7f80f406dfe0 2024-03-29T11:11:06.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.523+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80f80504d0 con 0x7f81080a9750 2024-03-29T11:11:06.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.523+0000 7f80effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:06.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.523+0000 7f80effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.841+0000 7f810cf5a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.841+0000 7f810cf5a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.841+0000 7f810cf5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.841+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f810804f730 con 0x7f81080a9750 2024-03-29T11:11:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.844+0000 7f80effff640 1 -- 172.21.15.105:0/1339643731 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f80f8052400 con 0x7f81080a9750 2024-03-29T11:11:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.844+0000 7f80effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.844+0000 7f80effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:06.822 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:06.822 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.846+0000 7f810cf5a640 10 librados: watch_flush enter 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 10 librados: watch_flush exit 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 10 client.4320.objecter successfully canceled tick 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 >> v1:172.21.15.110:6816/1087970004 conn(0x7f80f406dfe0 legacy=0x7f80f40704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 10 monclient: shutdown 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.847+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 >> v1:172.21.15.105:6789/0 conn(0x7f81080a9750 legacy=0x7f8108124480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.849+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 shutdown_connections 2024-03-29T11:11:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.849+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 >> 172.21.15.105:0/1339643731 conn(0x7f81080673e0 msgr2=0x7f8108069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.849+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 shutdown_connections 2024-03-29T11:11:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.849+0000 7f810cf5a640 1 -- 172.21.15.105:0/1339643731 wait complete. 2024-03-29T11:11:06.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:06.849+0000 7f810cf5a640 1 librados: shutdown 2024-03-29T11:11:06.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:13.653 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:13.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.830+0000 7f49b468d640 10 monclient: get_monmap_and_config 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.830+0000 7f49b468d640 10 monclient: build_initial_monmap 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.830+0000 7f49b468d640 10 monclient: monmap: 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:13.831392+0000 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:13.831392+0000 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:13.809 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:13.810 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.832+0000 7f49b468d640 1 Processor -- start 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.832+0000 7f49b468d640 1 -- start start 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.832+0000 7f49b468d640 10 monclient: init 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-b con 0x7f49ac0b0860 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-a con 0x7f49ac13ef30 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-c con 0x7f49ac0af8e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0b0c40 con 0x7f49ac13ef30 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0aed10 con 0x7f49ac0af8e0 2024-03-29T11:11:13.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0afd50 con 0x7f49ac0b0860 2024-03-29T11:11:13.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:13.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.833+0000 7f49b468d640 10 monclient(hunting): authenticate will time out at 948.109648s 2024-03-29T11:11:13.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.837+0000 7f49b2c2a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f49ac0af8e0 0x7f49ac149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:57773/0 (socket says 172.21.15.105:57773) 2024-03-29T11:11:13.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.837+0000 7f49b2c2a640 1 -- 172.21.15.105:0/1529102755 learned_addr learned my addr 172.21.15.105:0/1529102755 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:13.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868267130 0 0) 0x7f49ac0aed10 con 0x7f49ac0af8e0 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _negotiate 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4998003520 con 0x7f49ac0af8e0 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136629391 0 0) 0x7f49ac0afd50 con 0x7f49ac0b0860 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _negotiate 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac0aed10 con 0x7f49ac0b0860 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402784831 0 0) 0x7f49ac0b0c40 con 0x7f49ac13ef30 2024-03-29T11:11:13.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _negotiate 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac0afd50 con 0x7f49ac13ef30 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556962223 0 0) 0x7f4998003520 con 0x7f49ac0af8e0 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 >> v1:172.21.15.105:6790/0 conn(0x7f49ac0b0860 legacy=0x7f49ac13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 >> v1:172.21.15.105:6789/0 conn(0x7f49ac13ef30 legacy=0x7f49ac0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: _finish_hunting 0 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 monclient: found mon.noname-c 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0ab1b0 con 0x7f49ac0af8e0 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: _finish_auth 0 2024-03-29T11:11:13.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: _check_auth_tickets 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f49a8001690 con 0x7f49ac0af8e0 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.838+0000 7f49b1427640 10 monclient: dump: 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:13.821 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a80017f0 con 0x7f49ac0af8e0 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 1 -- 172.21.15.105:0/1529102755 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f49a8001c00 con 0x7f49ac0af8e0 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.839+0000 7f49b1427640 10 monclient: dump: 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:13.822 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:13.823 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:13.823 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.841+0000 7f49b468d640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:11:13.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.841+0000 7f49b468d640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.841+0000 7f49b468d640 4 set_mon_vals no callback set 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 10 monclient: shutdown 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 1 -- 172.21.15.105:0/1529102755 >> v1:172.21.15.110:6789/0 conn(0x7f49ac0af8e0 legacy=0x7f49ac149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 1 -- 172.21.15.105:0/1529102755 shutdown_connections 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 1 -- 172.21.15.105:0/1529102755 >> 172.21.15.105:0/1529102755 conn(0x7f49ac06b660 msgr2=0x7f49ac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 1 -- 172.21.15.105:0/1529102755 shutdown_connections 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.842+0000 7f49b468d640 1 -- 172.21.15.105:0/1529102755 wait complete. 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.843+0000 7f49b468d640 10 monclient: build_initial_monmap 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.843+0000 7f49b468d640 10 monclient: monmap: 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:13.824 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:13.825 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:13.825 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:13.825 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:13.825 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.825 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 librados: starting msgr at 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 librados: starting objecter 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 Processor -- start 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 -- start start 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 librados: setting wanted keys 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 1 librados: calling monclient init 2024-03-29T11:11:13.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.844+0000 7f49b468d640 10 monclient: init 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-c con 0x7f49ac0af8e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-a con 0x7f49ac0b0860 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient(hunting): picked mon.noname-b con 0x7f49ac13ef30 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0709a0 con 0x7f49ac0b0860 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0706a0 con 0x7f49ac13ef30 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac1e1aa0 con 0x7f49ac0af8e0 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:13.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.847+0000 7f49b468d640 10 monclient(hunting): authenticate will time out at 948.123587s 2024-03-29T11:11:13.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.851+0000 7f49b2c2a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f49ac13ef30 0x7f49ac06ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:57879/0 (socket says 172.21.15.105:57879) 2024-03-29T11:11:13.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.851+0000 7f49b2c2a640 1 -- 172.21.15.105:0/655057111 learned_addr learned my addr 172.21.15.105:0/655057111 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:13.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.851+0000 7f49a2ffd640 10 client.?.objecter ms_handle_connect 0x7f49ac13ef30 2024-03-29T11:11:13.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.851+0000 7f49a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:13.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 client.?.objecter ms_handle_connect 0x7f49ac0b0860 2024-03-29T11:11:13.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:13.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 client.?.objecter ms_handle_connect 0x7f49ac0af8e0 2024-03-29T11:11:13.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:13.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488648360 0 0) 0x7f49ac0706a0 con 0x7f49ac13ef30 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4990004eb0 con 0x7f49ac13ef30 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244582643 0 0) 0x7f49ac1e1aa0 con 0x7f49ac0af8e0 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac0706a0 con 0x7f49ac0af8e0 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811300475 0 0) 0x7f4990004eb0 con 0x7f49ac13ef30 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49ac1e1aa0 con 0x7f49ac13ef30 2024-03-29T11:11:13.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f49a8001bc0 con 0x7f49ac13ef30 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.855+0000 7f49a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:13.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.856+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1769614663 0 0) 0x7f49ac1e1aa0 con 0x7f49ac13ef30 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 >> v1:172.21.15.105:6790/0 conn(0x7f49ac0af8e0 legacy=0x7f49ac1e1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 >> v1:172.21.15.105:6789/0 conn(0x7f49ac0b0860 legacy=0x7f49ac06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 1 monclient: found mon.b 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac1e1c30 con 0x7f49ac13ef30 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:13.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 10 monclient: _renew_subs 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac070ec0 con 0x7f49ac13ef30 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 10 monclient: _renew_subs 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:13.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0713f0 con 0x7f49ac13ef30 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.857+0000 7f49b468d640 1 librados: init done 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a8004a90 con 0x7f49ac13ef30 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f49a8004da0 con 0x7f49ac13ef30 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.858+0000 7f49a2ffd640 10 monclient: dump: 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:13.842 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.859+0000 7f49b0c26640 4 set_mon_vals no callback set 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.861+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f49a801ad70 con 0x7f49ac13ef30 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.861+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f49a8084ba0 con 0x7f49ac13ef30 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49b468d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49a2ffd640 10 client.4330.objecter ms_dispatch 0x7f49ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49a2ffd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49a2ffd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49b468d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49b468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49a2ffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:11:13.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4994005080 con 0x7f49ac13ef30 2024-03-29T11:11:13.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49b2429640 10 monclient: get_auth_request con 0x7f499006dab0 auth_method 0 2024-03-29T11:11:13.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.862+0000 7f49a2ffd640 10 client.4330.objecter ms_handle_connect 0x7f499006dab0 2024-03-29T11:11:13.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.870+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49a8050480 con 0x7f49ac13ef30 2024-03-29T11:11:13.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.870+0000 7f49a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:13.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:13.870+0000 7f49a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.203+0000 7f49b468d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.203+0000 7f49b468d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.203+0000 7f49b468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.203+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4994005330 con 0x7f49ac13ef30 2024-03-29T11:11:14.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.207+0000 7f49a2ffd640 1 -- 172.21.15.105:0/655057111 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f49a80523b0 con 0x7f49ac13ef30 2024-03-29T11:11:14.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.207+0000 7f49a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:14.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.207+0000 7f49a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:14.186 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:14.186 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.210+0000 7f49b468d640 10 librados: watch_flush enter 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.210+0000 7f49b468d640 10 librados: watch_flush exit 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.210+0000 7f49b468d640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.210+0000 7f49b468d640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.211+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 >> v1:172.21.15.110:6816/1087970004 conn(0x7f499006dab0 legacy=0x7f499006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.211+0000 7f49b468d640 10 monclient: shutdown 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.211+0000 7f49b468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.211+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 >> v1:172.21.15.110:6789/0 conn(0x7f49ac13ef30 legacy=0x7f49ac06ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.212+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 shutdown_connections 2024-03-29T11:11:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.212+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 >> 172.21.15.105:0/655057111 conn(0x7f49ac06b660 msgr2=0x7f49ac13ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:14.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.214+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 shutdown_connections 2024-03-29T11:11:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.215+0000 7f49b468d640 1 -- 172.21.15.105:0/655057111 wait complete. 2024-03-29T11:11:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:14.215+0000 7f49b468d640 1 librados: shutdown 2024-03-29T11:11:14.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:21.021 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.187+0000 7f3d7a40f640 10 monclient: get_monmap_and_config 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.187+0000 7f3d7a40f640 10 monclient: build_initial_monmap 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.187+0000 7f3d7a40f640 10 monclient: monmap: 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:21.188853+0000 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:21.188853+0000 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:21.166 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:21.167 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:21.167 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.191+0000 7f3d7a40f640 1 Processor -- start 2024-03-29T11:11:21.170 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.191+0000 7f3d7a40f640 1 -- start start 2024-03-29T11:11:21.170 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.191+0000 7f3d7a40f640 10 monclient: init 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.191+0000 7f3d7a40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.191+0000 7f3d7a40f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d741422b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d74142690 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d740b17d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d74142c80 con 0x7f3d74142690 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740aae40 con 0x7f3d740b17d0 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7413fbd0 con 0x7f3d741422b0 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:21.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.192+0000 7f3d7a40f640 10 monclient(hunting): authenticate will time out at 955.468052s 2024-03-29T11:11:21.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d737fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f3d74142690 0x7f3d740b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:60535/0 (socket says 172.21.15.105:60535) 2024-03-29T11:11:21.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d737fe640 1 -- 172.21.15.105:0/2275162125 learned_addr learned my addr 172.21.15.105:0/2275162125 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:21.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860002018 0 0) 0x7f3d74142c80 con 0x7f3d74142690 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c003520 con 0x7f3d74142690 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831904499 0 0) 0x7f3d7413fbd0 con 0x7f3d741422b0 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d74142c80 con 0x7f3d741422b0 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738183746 0 0) 0x7f3d5c003520 con 0x7f3d74142690 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 >> v1:172.21.15.105:6790/0 conn(0x7f3d741422b0 legacy=0x7f3d7413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 >> v1:172.21.15.110:6789/0 conn(0x7f3d740b17d0 legacy=0x7f3d740b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 monclient: found mon.noname-a 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74151940 con 0x7f3d74142690 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d7a40f640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d64001690 con 0x7f3d74142690 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d72ffd640 10 monclient: dump: 2024-03-29T11:11:21.177 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.196+0000 7f3d7a40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d64001870 con 0x7f3d74142690 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:21.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 1 -- 172.21.15.105:0/2275162125 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d64001c00 con 0x7f3d74142690 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d72ffd640 10 monclient: dump: 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d7a40f640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:21.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d7a40f640 4 set_mon_vals no callback set 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d7a40f640 10 monclient: shutdown 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d7a40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.197+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2275162125 >> v1:172.21.15.105:6789/0 conn(0x7f3d74142690 legacy=0x7f3d740b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.199+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2275162125 shutdown_connections 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.199+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2275162125 >> 172.21.15.105:0/2275162125 conn(0x7f3d74066a00 msgr2=0x7f3d74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.200+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2275162125 shutdown_connections 2024-03-29T11:11:21.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.201+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2275162125 wait complete. 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.202+0000 7f3d7a40f640 10 monclient: build_initial_monmap 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.202+0000 7f3d7a40f640 10 monclient: monmap: 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.202+0000 7f3d7a40f640 1 librados: starting msgr at 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.202+0000 7f3d7a40f640 1 librados: starting objecter 2024-03-29T11:11:21.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.202+0000 7f3d7a40f640 1 Processor -- start 2024-03-29T11:11:21.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.203+0000 7f3d7a40f640 1 -- start start 2024-03-29T11:11:21.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.203+0000 7f3d7a40f640 1 librados: setting wanted keys 2024-03-29T11:11:21.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.203+0000 7f3d7a40f640 1 librados: calling monclient init 2024-03-29T11:11:21.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.203+0000 7f3d7a40f640 10 monclient: init 2024-03-29T11:11:21.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:21.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:21.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d740b17d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:21.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d741422b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d74142690 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740b77b0 con 0x7f3d741422b0 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7406e0e0 con 0x7f3d740b17d0 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740b5980 con 0x7f3d74142690 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:21.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.204+0000 7f3d7a40f640 10 monclient(hunting): authenticate will time out at 955.480636s 2024-03-29T11:11:21.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.211+0000 7f3d73fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3d740b17d0 0x7f3d740b68a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:58109/0 (socket says 172.21.15.105:58109) 2024-03-29T11:11:21.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.211+0000 7f3d73fff640 1 -- 172.21.15.105:0/2089350258 learned_addr learned my addr 172.21.15.105:0/2089350258 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:21.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.211+0000 7f3d70ff9640 10 client.?.objecter ms_handle_connect 0x7f3d740b17d0 2024-03-29T11:11:21.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:21.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076522296 0 0) 0x7f3d7406e0e0 con 0x7f3d740b17d0 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d60004eb0 con 0x7f3d740b17d0 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 client.?.objecter ms_handle_connect 0x7f3d74142690 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.212+0000 7f3d70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984650950 0 0) 0x7f3d60004eb0 con 0x7f3d740b17d0 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d7406e0e0 con 0x7f3d740b17d0 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d68001a10 con 0x7f3d740b17d0 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:21.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.213+0000 7f3d70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683184421 0 0) 0x7f3d740b5980 con 0x7f3d74142690 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:21.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d60005680 con 0x7f3d74142690 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 831787193 0 0) 0x7f3d7406e0e0 con 0x7f3d740b17d0 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 >> v1:172.21.15.105:6790/0 conn(0x7f3d74142690 legacy=0x7f3d7413be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 >> v1:172.21.15.105:6789/0 conn(0x7f3d741422b0 legacy=0x7f3d741e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 monclient: found mon.b 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d741e41a0 con 0x7f3d740b17d0 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:21.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 10 monclient: _renew_subs 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740b6fb0 con 0x7f3d740b17d0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 10 monclient: _renew_subs 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7414f270 con 0x7f3d740b17d0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.216+0000 7f3d7a40f640 1 librados: init done 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d68001cd0 con 0x7f3d740b17d0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3d68001a10 con 0x7f3d740b17d0 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:21.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d70ff9640 10 monclient: dump: 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.217+0000 7f3d71ffb640 4 set_mon_vals no callback set 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.218+0000 7f3d7a40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.218+0000 7f3d7a40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:21.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.218+0000 7f3d7a40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:21.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.218+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d54005080 con 0x7f3d740b17d0 2024-03-29T11:11:21.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.219+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f3d680038c0 con 0x7f3d740b17d0 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.220+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f3d68084850 con 0x7f3d740b17d0 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.220+0000 7f3d70ff9640 10 client.4333.objecter ms_dispatch 0x7f3d74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.220+0000 7f3d70ff9640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.220+0000 7f3d70ff9640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.220+0000 7f3d70ff9640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.222+0000 7f3d737fe640 10 monclient: get_auth_request con 0x7f3d6006dfb0 auth_method 0 2024-03-29T11:11:21.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.222+0000 7f3d70ff9640 10 client.4333.objecter ms_handle_connect 0x7f3d6006dfb0 2024-03-29T11:11:21.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.227+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d68050130 con 0x7f3d740b17d0 2024-03-29T11:11:21.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.227+0000 7f3d70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:21.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.227+0000 7f3d70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:21.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.556+0000 7f3d7a40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:21.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.556+0000 7f3d7a40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.556+0000 7f3d7a40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.556+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d54005330 con 0x7f3d740b17d0 2024-03-29T11:11:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.557+0000 7f3d70ff9640 1 -- 172.21.15.105:0/2089350258 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f3d68052060 con 0x7f3d740b17d0 2024-03-29T11:11:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.557+0000 7f3d70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.557+0000 7f3d70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:21.537 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:21.537 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:21.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.560+0000 7f3d7a40f640 10 librados: watch_flush enter 2024-03-29T11:11:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.561+0000 7f3d7a40f640 10 librados: watch_flush exit 2024-03-29T11:11:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.561+0000 7f3d7a40f640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:11:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.561+0000 7f3d7a40f640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:11:21.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 >> v1:172.21.15.110:6816/1087970004 conn(0x7f3d6006dfb0 legacy=0x7f3d60070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 10 monclient: shutdown 2024-03-29T11:11:21.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:21.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 >> v1:172.21.15.110:6789/0 conn(0x7f3d740b17d0 legacy=0x7f3d740b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:21.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 shutdown_connections 2024-03-29T11:11:21.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.562+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 >> 172.21.15.105:0/2089350258 conn(0x7f3d74066a00 msgr2=0x7f3d740b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:21.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.563+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 shutdown_connections 2024-03-29T11:11:21.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.563+0000 7f3d7a40f640 1 -- 172.21.15.105:0/2089350258 wait complete. 2024-03-29T11:11:21.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:21.563+0000 7f3d7a40f640 1 librados: shutdown 2024-03-29T11:11:21.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:28.370 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.552+0000 7f23f019a640 10 monclient: get_monmap_and_config 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.552+0000 7f23f019a640 10 monclient: build_initial_monmap 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.552+0000 7f23f019a640 10 monclient: monmap: 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:28.553150+0000 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:28.553150+0000 2024-03-29T11:11:28.531 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.553+0000 7f23f019a640 1 Processor -- start 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.554+0000 7f23f019a640 1 -- start start 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.554+0000 7f23f019a640 10 monclient: init 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.554+0000 7f23f019a640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.554+0000 7f23f019a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-a con 0x7f23e80a9ac0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-b con 0x7f23e813ee90 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-c con 0x7f23e80af900 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80afd10 con 0x7f23e80a9ac0 2024-03-29T11:11:28.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80af6f0 con 0x7f23e80af900 2024-03-29T11:11:28.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80aac80 con 0x7f23e813ee90 2024-03-29T11:11:28.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:28.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.555+0000 7f23f019a640 10 monclient(hunting): authenticate will time out at 962.831072s 2024-03-29T11:11:28.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.559+0000 7f23ed735640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f23e813ee90 0x7f23e813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55337/0 (socket says 172.21.15.105:55337) 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.559+0000 7f23ed735640 1 -- 172.21.15.105:0/2595008522 learned_addr learned my addr 172.21.15.105:0/2595008522 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145086143 0 0) 0x7f23e80aac80 con 0x7f23e813ee90 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 10 monclient(hunting): _negotiate 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:11:28.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.560+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d4003520 con 0x7f23e813ee90 2024-03-29T11:11:28.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687694026 0 0) 0x7f23e80afd10 con 0x7f23e80a9ac0 2024-03-29T11:11:28.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient(hunting): _negotiate 2024-03-29T11:11:28.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:28.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e80aac80 con 0x7f23e80a9ac0 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984541821 0 0) 0x7f23d4003520 con 0x7f23e813ee90 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 >> v1:172.21.15.110:6789/0 conn(0x7f23e80af900 legacy=0x7f23e8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 >> v1:172.21.15.105:6789/0 conn(0x7f23e80a9ac0 legacy=0x7f23e80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: _finish_hunting 0 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 monclient: found mon.noname-b 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80aaf60 con 0x7f23e813ee90 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: _finish_auth 0 2024-03-29T11:11:28.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: _check_auth_tickets 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f23d8004c90 con 0x7f23e813ee90 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23ecf34640 10 monclient: dump: 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:28.541 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:28.542 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:28.542 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23f019a640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:11:28.542 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.561+0000 7f23f019a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d8004df0 con 0x7f23e813ee90 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 1 -- 172.21.15.105:0/2595008522 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f23d8001410 con 0x7f23e813ee90 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23ecf34640 10 monclient: dump: 2024-03-29T11:11:28.545 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23f019a640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23f019a640 4 set_mon_vals no callback set 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23f019a640 10 monclient: shutdown 2024-03-29T11:11:28.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23f019a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.562+0000 7f23f019a640 1 -- 172.21.15.105:0/2595008522 >> v1:172.21.15.105:6790/0 conn(0x7f23e813ee90 legacy=0x7f23e813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.563+0000 7f23f019a640 1 -- 172.21.15.105:0/2595008522 shutdown_connections 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.563+0000 7f23f019a640 1 -- 172.21.15.105:0/2595008522 >> 172.21.15.105:0/2595008522 conn(0x7f23e80b35e0 msgr2=0x7f23e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.564+0000 7f23f019a640 1 -- 172.21.15.105:0/2595008522 shutdown_connections 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.564+0000 7f23f019a640 1 -- 172.21.15.105:0/2595008522 wait complete. 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 10 monclient: build_initial_monmap 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 10 monclient: monmap: 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.547 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 librados: starting msgr at 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 librados: starting objecter 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 Processor -- start 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 -- start start 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 librados: setting wanted keys 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 1 librados: calling monclient init 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.565+0000 7f23f019a640 10 monclient: init 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-a con 0x7f23e80a9ac0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:28.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-c con 0x7f23e80af900 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient(hunting): picked mon.noname-b con 0x7f23e813ee90 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e8141ff0 con 0x7f23e80a9ac0 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e8140b10 con 0x7f23e813ee90 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e81d8e10 con 0x7f23e80af900 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:28.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.566+0000 7f23f019a640 10 monclient(hunting): authenticate will time out at 962.842850s 2024-03-29T11:11:28.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23edf36640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f23e80a9ac0 0x7f23e8071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:40145/0 (socket says 172.21.15.105:40145) 2024-03-29T11:11:28.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23edf36640 1 -- 172.21.15.105:0/2834301316 learned_addr learned my addr 172.21.15.105:0/2834301316 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 client.?.objecter ms_handle_connect 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858141221 0 0) 0x7f23e8141ff0 con 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc004eb0 con 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3735457672 0 0) 0x7f23cc004eb0 con 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23e8141ff0 con 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f23e4001a10 con 0x7f23e80a9ac0 2024-03-29T11:11:28.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.570+0000 7f23de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 client.?.objecter ms_handle_connect 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294778921 0 0) 0x7f23e81d8e10 con 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc004eb0 con 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790498789 0 0) 0x7f23cc004eb0 con 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23cc0059e0 con 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f23d80013b0 con 0x7f23e80af900 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:28.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.571+0000 7f23de7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 236706780 0 0) 0x7f23cc0059e0 con 0x7f23e80af900 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 >> v1:172.21.15.110:6789/0 conn(0x7f23e813ee90 legacy=0x7f23e81db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 >> v1:172.21.15.105:6789/0 conn(0x7f23e80a9ac0 legacy=0x7f23e8071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 monclient: found mon.c 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e813f920 con 0x7f23e80af900 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 10 monclient: _renew_subs 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e81423a0 con 0x7f23e80af900 2024-03-29T11:11:28.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 10 monclient: _renew_subs 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e8072510 con 0x7f23e80af900 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23f019a640 1 librados: init done 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d80017e0 con 0x7f23e80af900 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f23d8001c30 con 0x7f23e80af900 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23de7fc640 10 monclient: dump: 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:28.560 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.572+0000 7f23dffff640 4 set_mon_vals no callback set 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.574+0000 7f23f019a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.574+0000 7f23f019a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.574+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f23d801add0 con 0x7f23e80af900 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.574+0000 7f23f019a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.574+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d0005080 con 0x7f23e80af900 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f23d8084900 con 0x7f23e80af900 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 10 client.4370.objecter ms_dispatch 0x7f23e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23edf36640 10 monclient: get_auth_request con 0x7f23cc06e090 auth_method 0 2024-03-29T11:11:28.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.575+0000 7f23de7fc640 10 client.4370.objecter ms_handle_connect 0x7f23cc06e090 2024-03-29T11:11:28.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.584+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23d80501e0 con 0x7f23e80af900 2024-03-29T11:11:28.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.584+0000 7f23de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:28.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.584+0000 7f23de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:28.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.900+0000 7f23f019a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:28.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.900+0000 7f23f019a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:28.879 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.900+0000 7f23f019a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:28.879 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.900+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d0005330 con 0x7f23e80af900 2024-03-29T11:11:28.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.903+0000 7f23de7fc640 1 -- 172.21.15.105:0/2834301316 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f23d80501e0 con 0x7f23e80af900 2024-03-29T11:11:28.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.903+0000 7f23de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:28.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.903+0000 7f23de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:28.881 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:28.882 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:28.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.906+0000 7f23f019a640 10 librados: watch_flush enter 2024-03-29T11:11:28.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 10 librados: watch_flush exit 2024-03-29T11:11:28.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 >> v1:172.21.15.110:6816/1087970004 conn(0x7f23cc06e090 legacy=0x7f23cc070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 10 monclient: shutdown 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 >> v1:172.21.15.105:6790/0 conn(0x7f23e80af900 legacy=0x7f23e81db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 shutdown_connections 2024-03-29T11:11:28.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.907+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 >> 172.21.15.105:0/2834301316 conn(0x7f23e80b35e0 msgr2=0x7f23e80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:28.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.908+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 shutdown_connections 2024-03-29T11:11:28.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.908+0000 7f23f019a640 1 -- 172.21.15.105:0/2834301316 wait complete. 2024-03-29T11:11:28.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:28.911+0000 7f23f019a640 1 librados: shutdown 2024-03-29T11:11:28.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:35.712 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.881+0000 7fd82e90b640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.881+0000 7fd82e90b640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.881+0000 7fd82e90b640 10 monclient: monmap: 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:35.881932+0000 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:35.881932+0000 2024-03-29T11:11:35.859 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.860 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:35.860 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:35.860 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:35.860 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:35.860 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:35.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.885+0000 7fd82e90b640 1 Processor -- start 2024-03-29T11:11:35.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.886+0000 7fd82e90b640 1 -- start start 2024-03-29T11:11:35.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.886+0000 7fd82e90b640 10 monclient: init 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-b con 0x7fd8200389e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-a con 0x7fd820038dc0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-c con 0x7fd820030660 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200c54c0 con 0x7fd820038dc0 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820036430 con 0x7fd820030660 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820039840 con 0x7fd8200389e0 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.887+0000 7fd82e90b640 10 monclient(hunting): authenticate will time out at 970.162645s 2024-03-29T11:11:35.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.891+0000 7fd82e10a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd820030660 0x7fd8200d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:42285/0 (socket says 172.21.15.105:42285) 2024-03-29T11:11:35.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.891+0000 7fd82e10a640 1 -- 172.21.15.105:0/4234851838 learned_addr learned my addr 172.21.15.105:0/4234851838 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:35.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.893+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678192502 0 0) 0x7fd820036430 con 0x7fd820030660 2024-03-29T11:11:35.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.893+0000 7fd82c907640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.893+0000 7fd82c907640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.893+0000 7fd82c907640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.893+0000 7fd82c907640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:11:35.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd814003520 con 0x7fd820030660 2024-03-29T11:11:35.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587285985 0 0) 0x7fd814003520 con 0x7fd820030660 2024-03-29T11:11:35.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 >> v1:172.21.15.105:6790/0 conn(0x7fd8200389e0 legacy=0x7fd820033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:35.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 >> v1:172.21.15.105:6789/0 conn(0x7fd820038dc0 legacy=0x7fd8200cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:35.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820035690 con 0x7fd820030660 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82e90b640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82e90b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fd818001690 con 0x7fd820030660 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.878 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: dump: 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:35.879 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82e90b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 1 -- 172.21.15.105:0/4234851838 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd818001920 con 0x7fd820030660 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82c907640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82e90b640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.894+0000 7fd82e90b640 4 set_mon_vals no callback set 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.895+0000 7fd82e90b640 10 monclient: shutdown 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.895+0000 7fd82e90b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.895+0000 7fd82e90b640 1 -- 172.21.15.105:0/4234851838 >> v1:172.21.15.110:6789/0 conn(0x7fd820030660 legacy=0x7fd8200d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.895+0000 7fd82e90b640 1 -- 172.21.15.105:0/4234851838 shutdown_connections 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.895+0000 7fd82e90b640 1 -- 172.21.15.105:0/4234851838 >> 172.21.15.105:0/4234851838 conn(0x7fd820005c00 msgr2=0x7fd820006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.896+0000 7fd82e90b640 1 -- 172.21.15.105:0/4234851838 shutdown_connections 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.896+0000 7fd82e90b640 1 -- 172.21.15.105:0/4234851838 wait complete. 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.880 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 10 monclient: monmap: 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 librados: starting msgr at 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 librados: starting objecter 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 Processor -- start 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 -- start start 2024-03-29T11:11:35.881 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 librados: setting wanted keys 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 1 librados: calling monclient init 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.897+0000 7fd82e90b640 10 monclient: init 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-c con 0x7fd820030660 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-b con 0x7fd8200389e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient(hunting): picked mon.noname-a con 0x7fd8200c6a60 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820158310 con 0x7fd8200c6a60 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200c56c0 con 0x7fd8200389e0 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200c8bd0 con 0x7fd820030660 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.898+0000 7fd82e90b640 10 monclient(hunting): authenticate will time out at 970.173626s 2024-03-29T11:11:35.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd82d909640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fd820030660 0x7fd8201587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:48379/0 (socket says 172.21.15.105:48379) 2024-03-29T11:11:35.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd82d909640 1 -- 172.21.15.105:0/929763334 learned_addr learned my addr 172.21.15.105:0/929763334 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:35.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 client.?.objecter ms_handle_connect 0x7fd820030660 2024-03-29T11:11:35.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123804031 0 0) 0x7fd8200c8bd0 con 0x7fd820030660 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.903+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c004eb0 con 0x7fd820030660 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 936855908 0 0) 0x7fd80c004eb0 con 0x7fd820030660 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8200c8bd0 con 0x7fd820030660 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fd82804fdd0 con 0x7fd820030660 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.887 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057884047 0 0) 0x7fd8200c8bd0 con 0x7fd820030660 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.904+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 >> v1:172.21.15.110:6789/0 conn(0x7fd8200389e0 legacy=0x7fd8200c6350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.905+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 >> v1:172.21.15.105:6789/0 conn(0x7fd8200c6a60 legacy=0x7fd8200be170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.905+0000 7fd81e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.905+0000 7fd81e7fc640 1 monclient: found mon.c 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.905+0000 7fd81e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.905+0000 7fd81e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:35.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200c8d60 con 0x7fd820030660 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 client.?.objecter ms_handle_connect 0x7fd8200c6a60 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd82804f400 con 0x7fd820030660 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fd828075710 con 0x7fd820030660 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81e7fc640 10 monclient: dump: 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.889 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.906+0000 7fd81ffff640 4 set_mon_vals no callback set 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 10 monclient: _renew_subs 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd820158eb0 con 0x7fd820030660 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:35.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 10 monclient: _renew_subs 2024-03-29T11:11:35.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:35.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8200c85d0 con 0x7fd820030660 2024-03-29T11:11:35.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.908+0000 7fd82e90b640 1 librados: init done 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.910+0000 7fd82e90b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.910+0000 7fd82e90b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.910+0000 7fd82e90b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.910+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd820007b90 con 0x7fd820030660 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fd828067bf0 con 0x7fd820030660 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fd8280ff590 con 0x7fd820030660 2024-03-29T11:11:35.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 10 client.4373.objecter ms_dispatch 0x7fd828000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:35.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:35.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:35.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.914+0000 7fd81e7fc640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:11:35.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.915+0000 7fd82d108640 10 monclient: get_auth_request con 0x7fd80c06da90 auth_method 0 2024-03-29T11:11:35.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.915+0000 7fd81e7fc640 10 client.4373.objecter ms_handle_connect 0x7fd80c06da90 2024-03-29T11:11:35.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.922+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8280cae70 con 0x7fd820030660 2024-03-29T11:11:35.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.922+0000 7fd81e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:35.922+0000 7fd81e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:36.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.242+0000 7fd82e90b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.242+0000 7fd82e90b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.242+0000 7fd82e90b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:11:36.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.242+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd820005120 con 0x7fd820030660 2024-03-29T11:11:36.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.245+0000 7fd81e7fc640 1 -- 172.21.15.105:0/929763334 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fd8280ccda0 con 0x7fd820030660 2024-03-29T11:11:36.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.245+0000 7fd81e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.245+0000 7fd81e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:36.223 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:36.224 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.248+0000 7fd82e90b640 10 librados: watch_flush enter 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.248+0000 7fd82e90b640 10 librados: watch_flush exit 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.248+0000 7fd82e90b640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.248+0000 7fd82e90b640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.249+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 >> v1:172.21.15.110:6816/1087970004 conn(0x7fd80c06da90 legacy=0x7fd80c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.249+0000 7fd82e90b640 10 monclient: shutdown 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.249+0000 7fd82e90b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:36.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.249+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 >> v1:172.21.15.105:6790/0 conn(0x7fd820030660 legacy=0x7fd8201587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.251+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 shutdown_connections 2024-03-29T11:11:36.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.251+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 >> 172.21.15.105:0/929763334 conn(0x7fd820005c00 msgr2=0x7fd820034920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.252+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 shutdown_connections 2024-03-29T11:11:36.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.252+0000 7fd82e90b640 1 -- 172.21.15.105:0/929763334 wait complete. 2024-03-29T11:11:36.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:36.252+0000 7fd82e90b640 1 librados: shutdown 2024-03-29T11:11:36.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:43.055 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:43.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.231+0000 7f9a7dce2640 10 monclient: get_monmap_and_config 2024-03-29T11:11:43.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.231+0000 7f9a7dce2640 10 monclient: build_initial_monmap 2024-03-29T11:11:43.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.231+0000 7f9a7dce2640 10 monclient: monmap: 2024-03-29T11:11:43.209 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:43.209 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:43.232307+0000 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:43.232307+0000 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:43.210 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.232+0000 7f9a7dce2640 1 Processor -- start 2024-03-29T11:11:43.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.232+0000 7f9a7dce2640 1 -- start start 2024-03-29T11:11:43.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.232+0000 7f9a7dce2640 10 monclient: init 2024-03-29T11:11:43.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:43.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-a con 0x7f9a780b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7814a700 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-c con 0x7f9a780ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780aba70 con 0x7f9a780b0770 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780aecb0 con 0x7f9a780ac5a0 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780af150 con 0x7f9a7814a700 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:43.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.233+0000 7f9a7dce2640 10 monclient(hunting): authenticate will time out at 977.509947s 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a77fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9a780ac5a0 0x7f9a78151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:37600/0 (socket says 172.21.15.105:37600) 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a77fff640 1 -- 172.21.15.105:0/1972640794 learned_addr learned my addr 172.21.15.105:0/1972640794 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469368080 0 0) 0x7f9a780aecb0 con 0x7f9a780ac5a0 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.234+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003520 con 0x7f9a780ac5a0 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898833375 0 0) 0x7f9a780aba70 con 0x7f9a780b0770 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a780aecb0 con 0x7f9a780b0770 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117695643 0 0) 0x7f9a780af150 con 0x7f9a7814a700 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a780aba70 con 0x7f9a7814a700 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3187533532 0 0) 0x7f9a780aecb0 con 0x7f9a780b0770 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 >> v1:172.21.15.105:6790/0 conn(0x7f9a7814a700 legacy=0x7f9a780abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 >> v1:172.21.15.110:6789/0 conn(0x7f9a780ac5a0 legacy=0x7f9a78151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 monclient: found mon.noname-a 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:43.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780ac9d0 con 0x7f9a780b0770 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9a68001670 con 0x7f9a780b0770 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: dump: 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:43.216 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a7dce2640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a7dce2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a680017d0 con 0x7f9a780b0770 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 1 -- 172.21.15.105:0/1972640794 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9a68001c20 con 0x7f9a780b0770 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.235+0000 7f9a76ffd640 10 monclient: dump: 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.217 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.236+0000 7f9a7dce2640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:43.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.236+0000 7f9a7dce2640 4 set_mon_vals no callback set 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.236+0000 7f9a7dce2640 10 monclient: shutdown 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.236+0000 7f9a7dce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.236+0000 7f9a7dce2640 1 -- 172.21.15.105:0/1972640794 >> v1:172.21.15.105:6789/0 conn(0x7f9a780b0770 legacy=0x7f9a78149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.237+0000 7f9a7dce2640 1 -- 172.21.15.105:0/1972640794 shutdown_connections 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.237+0000 7f9a7dce2640 1 -- 172.21.15.105:0/1972640794 >> 172.21.15.105:0/1972640794 conn(0x7f9a7806b660 msgr2=0x7f9a7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.237+0000 7f9a7dce2640 1 -- 172.21.15.105:0/1972640794 shutdown_connections 2024-03-29T11:11:43.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.237+0000 7f9a7dce2640 1 -- 172.21.15.105:0/1972640794 wait complete. 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.238+0000 7f9a7dce2640 10 monclient: build_initial_monmap 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.238+0000 7f9a7dce2640 10 monclient: monmap: 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.238+0000 7f9a7dce2640 1 librados: starting msgr at 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.238+0000 7f9a7dce2640 1 librados: starting objecter 2024-03-29T11:11:43.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.238+0000 7f9a7dce2640 1 Processor -- start 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.239+0000 7f9a7dce2640 1 -- start start 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.239+0000 7f9a7dce2640 1 librados: setting wanted keys 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.239+0000 7f9a7dce2640 1 librados: calling monclient init 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.239+0000 7f9a7dce2640 10 monclient: init 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-b con 0x7f9a780ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-c con 0x7f9a780b0770 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7814a700 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a7dce2640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a78147ae0 con 0x7f9a7814a700 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a6ffff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f9a780b0770 0x7f9a781db690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:56886/0 (socket says 172.21.15.105:56886) 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a6ffff640 1 -- 172.21.15.105:0/784880340 learned_addr learned my addr 172.21.15.105:0/784880340 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a74ff9640 10 client.?.objecter ms_handle_connect 0x7f9a780b0770 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:43.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a74ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7814a700 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.240+0000 7f9a74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291938881 0 0) 0x7f9a78147ae0 con 0x7f9a7814a700 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a78147c70 con 0x7f9a780ac5a0 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a78128140 con 0x7f9a780b0770 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a7dce2640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a7dce2640 10 monclient(hunting): authenticate will time out at 977.517322s 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a58004eb0 con 0x7f9a7814a700 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028500614 0 0) 0x7f9a78128140 con 0x7f9a780b0770 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.241+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78147ae0 con 0x7f9a780b0770 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 client.?.objecter ms_handle_connect 0x7f9a780ac5a0 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046821411 0 0) 0x7f9a58004eb0 con 0x7f9a7814a700 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a78128140 con 0x7f9a7814a700 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 725494374 0 0) 0x7f9a78147ae0 con 0x7f9a780b0770 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a58004eb0 con 0x7f9a780b0770 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9a70001c70 con 0x7f9a7814a700 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:43.230 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94562112 0 0) 0x7f9a78147c70 con 0x7f9a780ac5a0 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.242+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78147ae0 con 0x7f9a780ac5a0 2024-03-29T11:11:43.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9a600018d0 con 0x7f9a780b0770 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 88094034 0 0) 0x7f9a78128140 con 0x7f9a7814a700 2024-03-29T11:11:43.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 >> v1:172.21.15.105:6790/0 conn(0x7f9a780b0770 legacy=0x7f9a781db690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 >> v1:172.21.15.110:6789/0 conn(0x7f9a780ac5a0 legacy=0x7f9a78128b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 monclient: found mon.a 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a781282d0 con 0x7f9a7814a700 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 10 monclient: _renew_subs 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a781dbda0 con 0x7f9a7814a700 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:43.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 10 monclient: _renew_subs 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a781dc230 con 0x7f9a7814a700 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.243+0000 7f9a7dce2640 1 librados: init done 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a70004b60 con 0x7f9a7814a700 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9a70004fb0 con 0x7f9a7814a700 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a767fc640 4 set_mon_vals no callback set 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.244+0000 7f9a74ff9640 10 monclient: dump: 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:43.234 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.245+0000 7f9a7dce2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.245+0000 7f9a7dce2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.245+0000 7f9a7dce2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.245+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c005080 con 0x7f9a7814a700 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.246+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f9a7001af80 con 0x7f9a7814a700 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a777fe640 10 monclient: get_auth_request con 0x7f9a5806e080 auth_method 0 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f9a70084be0 con 0x7f9a7814a700 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 10 client.4341.objecter ms_dispatch 0x7f9a78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:43.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:43.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:11:43.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.247+0000 7f9a74ff9640 10 client.4341.objecter ms_handle_connect 0x7f9a5806e080 2024-03-29T11:11:43.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.258+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a700504c0 con 0x7f9a7814a700 2024-03-29T11:11:43.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.258+0000 7f9a74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:43.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.258+0000 7f9a74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:43.603 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.625+0000 7f9a7dce2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:43.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.625+0000 7f9a7dce2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:43.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.625+0000 7f9a7dce2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:43.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.625+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a5c005330 con 0x7f9a7814a700 2024-03-29T11:11:43.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.626+0000 7f9a74ff9640 1 -- 172.21.15.105:0/784880340 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f9a700523f0 con 0x7f9a7814a700 2024-03-29T11:11:43.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.626+0000 7f9a74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:43.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.626+0000 7f9a74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:43.606 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:43.606 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.629+0000 7f9a7dce2640 10 librados: watch_flush enter 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.629+0000 7f9a7dce2640 10 librados: watch_flush exit 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.629+0000 7f9a7dce2640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.629+0000 7f9a7dce2640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 >> v1:172.21.15.110:6816/1087970004 conn(0x7f9a5806e080 legacy=0x7f9a58070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 10 monclient: shutdown 2024-03-29T11:11:43.610 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:43.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 >> v1:172.21.15.105:6789/0 conn(0x7f9a7814a700 legacy=0x7f9a781e5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:43.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 shutdown_connections 2024-03-29T11:11:43.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 >> 172.21.15.105:0/784880340 conn(0x7f9a7806b660 msgr2=0x7f9a78071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:43.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 shutdown_connections 2024-03-29T11:11:43.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 -- 172.21.15.105:0/784880340 wait complete. 2024-03-29T11:11:43.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:43.630+0000 7f9a7dce2640 1 librados: shutdown 2024-03-29T11:11:43.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:50.435 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.619+0000 7fb9ac7b5640 10 monclient: get_monmap_and_config 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.619+0000 7fb9ac7b5640 10 monclient: build_initial_monmap 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.619+0000 7fb9ac7b5640 10 monclient: monmap: 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:50.620099+0000 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:50.620099+0000 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:50.597 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.598 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:50.598 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:50.598 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:50.598 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.623+0000 7fb9ac7b5640 1 Processor -- start 2024-03-29T11:11:50.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.623+0000 7fb9ac7b5640 1 -- start start 2024-03-29T11:11:50.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.623+0000 7fb9ac7b5640 10 monclient: init 2024-03-29T11:11:50.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a40b1f60 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a4071eb0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a4143d10 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a406ef80 con 0x7fb9a4071eb0 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a406f260 con 0x7fb9a4143d10 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40b40c0 con 0x7fb9a40b1f60 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.624+0000 7fb9ac7b5640 10 monclient(hunting): authenticate will time out at 984.900553s 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.627+0000 7fb9aad52640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb9a4143d10 0x7fb9a4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:45343/0 (socket says 172.21.15.105:45343) 2024-03-29T11:11:50.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.627+0000 7fb9aad52640 1 -- 172.21.15.105:0/1603891518 learned_addr learned my addr 172.21.15.105:0/1603891518 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078724720 0 0) 0x7fb9a40b40c0 con 0x7fb9a40b1f60 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 10 monclient(hunting): _negotiate 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:11:50.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.630+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990003520 con 0x7fb9a40b1f60 2024-03-29T11:11:50.612 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012476996 0 0) 0x7fb990003520 con 0x7fb9a40b1f60 2024-03-29T11:11:50.612 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 >> v1:172.21.15.110:6789/0 conn(0x7fb9a4143d10 legacy=0x7fb9a4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 >> v1:172.21.15.105:6789/0 conn(0x7fb9a4071eb0 legacy=0x7fb9a4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: _finish_hunting 0 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 monclient: found mon.noname-b 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4155d70 con 0x7fb9a40b1f60 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: _finish_auth 0 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: _check_auth_tickets 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb994001690 con 0x7fb9a40b1f60 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.632+0000 7fb9a954f640 10 monclient: dump: 2024-03-29T11:11:50.613 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.634+0000 7fb9ac7b5640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:11:50.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.634+0000 7fb9ac7b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9940017f0 con 0x7fb9a40b1f60 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 1 -- 172.21.15.105:0/1603891518 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb994001c40 con 0x7fb9a40b1f60 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.636+0000 7fb9a954f640 10 monclient: dump: 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.615 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.637+0000 7fb9ac7b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.637+0000 7fb9ac7b5640 4 set_mon_vals no callback set 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.638+0000 7fb9ac7b5640 10 monclient: shutdown 2024-03-29T11:11:50.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.638+0000 7fb9ac7b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:50.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.638+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/1603891518 >> v1:172.21.15.105:6790/0 conn(0x7fb9a40b1f60 legacy=0x7fb9a4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.618 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.641+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/1603891518 shutdown_connections 2024-03-29T11:11:50.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.641+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/1603891518 >> 172.21.15.105:0/1603891518 conn(0x7fb9a4066a00 msgr2=0x7fb9a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:50.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.641+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/1603891518 shutdown_connections 2024-03-29T11:11:50.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.641+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/1603891518 wait complete. 2024-03-29T11:11:50.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 10 monclient: build_initial_monmap 2024-03-29T11:11:50.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 10 monclient: monmap: 2024-03-29T11:11:50.621 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 librados: starting msgr at 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 librados: starting objecter 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 Processor -- start 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 -- start start 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 librados: setting wanted keys 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 1 librados: calling monclient init 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.642+0000 7fb9ac7b5640 10 monclient: init 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a40b1f60 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:50.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a4143d10 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a4151df0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a41479c0 con 0x7fb9a4143d10 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a413f810 con 0x7fb9a4151df0 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4148560 con 0x7fb9a40b1f60 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:50.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.643+0000 7fb9ac7b5640 10 monclient(hunting): authenticate will time out at 984.919647s 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.648+0000 7fb9a9d50640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fb9a4143d10 0x7fb9a413e9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:51259/0 (socket says 172.21.15.105:51259) 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.648+0000 7fb9a9d50640 1 -- 172.21.15.105:0/3630619611 learned_addr learned my addr 172.21.15.105:0/3630619611 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.648+0000 7fb99affd640 10 client.?.objecter ms_handle_connect 0x7fb9a4143d10 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048990158 0 0) 0x7fb9a41479c0 con 0x7fb9a4143d10 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:50.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb988004eb0 con 0x7fb9a4143d10 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 client.?.objecter ms_handle_connect 0x7fb9a4151df0 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481845299 0 0) 0x7fb988004eb0 con 0x7fb9a4143d10 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a41479c0 con 0x7fb9a4143d10 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb9a0001a10 con 0x7fb9a4143d10 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:50.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): dump: 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.649+0000 7fb99affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:50.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 client.?.objecter ms_handle_connect 0x7fb9a40b1f60 2024-03-29T11:11:50.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:50.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003935105 0 0) 0x7fb9a413f810 con 0x7fb9a4151df0 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb988004eb0 con 0x7fb9a4151df0 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34703187 0 0) 0x7fb9a4148560 con 0x7fb9a40b1f60 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a413f810 con 0x7fb9a40b1f60 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027822337 0 0) 0x7fb988004eb0 con 0x7fb9a4151df0 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a4148560 con 0x7fb9a4151df0 2024-03-29T11:11:50.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb99c001a90 con 0x7fb9a4151df0 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.653+0000 7fb99affd640 10 monclient(hunting): dump: 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064571516 0 0) 0x7fb9a413f810 con 0x7fb9a40b1f60 2024-03-29T11:11:50.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb988004eb0 con 0x7fb9a40b1f60 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1057367230 0 0) 0x7fb9a4148560 con 0x7fb9a4151df0 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 >> v1:172.21.15.105:6790/0 conn(0x7fb9a40b1f60 legacy=0x7fb9a4147e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 >> v1:172.21.15.105:6789/0 conn(0x7fb9a4143d10 legacy=0x7fb9a413e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 monclient: found mon.b 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a414b4d0 con 0x7fb9a4151df0 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 10 monclient: _finish_auth 0 2024-03-29T11:11:50.637 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb99affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 10 monclient: _renew_subs 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a4152890 con 0x7fb9a4151df0 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 10 monclient: _renew_subs 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.654+0000 7fb9ac7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.655+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a41549b0 con 0x7fb9a4151df0 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.655+0000 7fb9ac7b5640 1 librados: init done 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.656+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb99c001ee0 con 0x7fb9a4151df0 2024-03-29T11:11:50.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.656+0000 7fb99affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.656+0000 7fb99bfff640 4 set_mon_vals no callback set 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.657+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb99c004b60 con 0x7fb9a4151df0 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb9ac7b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb9ac7b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb9ac7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:50.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98c005080 con 0x7fb9a4151df0 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb99affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb99affd640 10 monclient: dump: 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:50.641 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.658+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb99c004d80 con 0x7fb9a4151df0 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fb99c084ba0 con 0x7fb9a4151df0 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb99affd640 10 client.4348.objecter ms_dispatch 0x7fb9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb99affd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb99affd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb99affd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:11:50.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.659+0000 7fb9aa551640 10 monclient: get_auth_request con 0x7fb98806e280 auth_method 0 2024-03-29T11:11:50.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.662+0000 7fb99affd640 10 client.4348.objecter ms_handle_connect 0x7fb98806e280 2024-03-29T11:11:50.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.665+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb99c0504a0 con 0x7fb9a4151df0 2024-03-29T11:11:50.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.666+0000 7fb99affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:50.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:50.666+0000 7fb99affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:51.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.069+0000 7fb9ac7b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:51.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.069+0000 7fb9ac7b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:51.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.069+0000 7fb9ac7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:11:51.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.069+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb98c005330 con 0x7fb9a4151df0 2024-03-29T11:11:51.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.070+0000 7fb99affd640 1 -- 172.21.15.105:0/3630619611 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fb99c0523d0 con 0x7fb9a4151df0 2024-03-29T11:11:51.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.070+0000 7fb99affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:51.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.070+0000 7fb99affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:51.048 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:51.048 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 10 librados: watch_flush enter 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 10 librados: watch_flush exit 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb98806e280 legacy=0x7fb988070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 10 monclient: shutdown 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:51.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 >> v1:172.21.15.110:6789/0 conn(0x7fb9a4151df0 legacy=0x7fb9a413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 shutdown_connections 2024-03-29T11:11:51.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.074+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 >> 172.21.15.105:0/3630619611 conn(0x7fb9a4066a00 msgr2=0x7fb9a4066e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:51.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.075+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 shutdown_connections 2024-03-29T11:11:51.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.075+0000 7fb9ac7b5640 1 -- 172.21.15.105:0/3630619611 wait complete. 2024-03-29T11:11:51.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:51.075+0000 7fb9ac7b5640 1 librados: shutdown 2024-03-29T11:11:51.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:57.876 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.049+0000 7f62720d9640 10 monclient: get_monmap_and_config 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.049+0000 7f62720d9640 10 monclient: build_initial_monmap 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.049+0000 7f62720d9640 10 monclient: monmap: 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:11:58.050955+0000 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:11:58.050955+0000 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.028 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.051+0000 7f62720d9640 1 Processor -- start 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.051+0000 7f62720d9640 1 -- start start 2024-03-29T11:11:58.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.051+0000 7f62720d9640 10 monclient: init 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.052+0000 7f62720d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.052+0000 7f62720d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.052+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-a con 0x7f626c1457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-c con 0x7f626c13f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626b7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f626c1457f0 0x7f626c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:36732/0 (socket says 172.21.15.105:36732) 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626b7fe640 1 -- 172.21.15.105:0/1079485466 learned_addr learned my addr 172.21.15.105:0/1079485466 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-b con 0x7f626c0a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0aaef0 con 0x7f626c1457f0 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c145bd0 con 0x7f626c13f650 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c145eb0 con 0x7f626c0a9d70 2024-03-29T11:11:58.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f62720d9640 10 monclient(hunting): authenticate will time out at 992.329280s 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694364640 0 0) 0x7f626c145bd0 con 0x7f626c13f650 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258003520 con 0x7f626c13f650 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.053+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569775278 0 0) 0x7f626c0aaef0 con 0x7f626c1457f0 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c145bd0 con 0x7f626c1457f0 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037571372 0 0) 0x7f6258003520 con 0x7f626c13f650 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 >> v1:172.21.15.105:6790/0 conn(0x7f626c0a9d70 legacy=0x7f626c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 >> v1:172.21.15.105:6789/0 conn(0x7f626c1457f0 legacy=0x7f626c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 monclient: found mon.noname-c 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c146340 con 0x7f626c13f650 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: _finish_auth 0 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:58.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6254001690 con 0x7f626c13f650 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: dump: 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f62720d9640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f62720d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62540017f0 con 0x7f626c13f650 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 1 -- 172.21.15.105:0/1079485466 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6254001c00 con 0x7f626c13f650 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f626affd640 10 monclient: dump: 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.038 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f62720d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.054+0000 7f62720d9640 4 set_mon_vals no callback set 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.055+0000 7f62720d9640 10 monclient: shutdown 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.055+0000 7f62720d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.055+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 >> v1:172.21.15.110:6789/0 conn(0x7f626c13f650 legacy=0x7f626c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.055+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 shutdown_connections 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.056+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 >> 172.21.15.105:0/1079485466 conn(0x7f626c066a00 msgr2=0x7f626c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.056+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 shutdown_connections 2024-03-29T11:11:58.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.056+0000 7f62720d9640 1 -- 172.21.15.105:0/1079485466 wait complete. 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.057+0000 7f62720d9640 10 monclient: build_initial_monmap 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.057+0000 7f62720d9640 10 monclient: monmap: 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:11:58.044 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.057+0000 7f62720d9640 1 librados: starting msgr at 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 1 librados: starting objecter 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 1 Processor -- start 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 1 -- start start 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 1 librados: setting wanted keys 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 1 librados: calling monclient init 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.058+0000 7f62720d9640 10 monclient: init 2024-03-29T11:11:58.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-a con 0x7f626c0a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-c con 0x7f626c13f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f626b7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f626c0a9d70 0x7f626c13b750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:37026/0 (socket says 172.21.15.105:37026) 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f626b7fe640 1 -- 172.21.15.105:0/3028040201 learned_addr learned my addr 172.21.15.105:0/3028040201 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f6268ff9640 10 client.?.objecter ms_handle_connect 0x7f626c0a9d70 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f6268ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 10 monclient(hunting): picked mon.noname-b con 0x7f626c1457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.059+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c13afc0 con 0x7f626c0a9d70 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c138df0 con 0x7f626c1457f0 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c13ace0 con 0x7f626c13f650 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f62720d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f62720d9640 10 monclient(hunting): authenticate will time out at 992.336099s 2024-03-29T11:11:58.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 client.?.objecter ms_handle_connect 0x7f626c13f650 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 client.?.objecter ms_handle_connect 0x7f626c1457f0 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979937683 0 0) 0x7f626c13afc0 con 0x7f626c0a9d70 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c004eb0 con 0x7f626c0a9d70 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1479919807 0 0) 0x7f624c004eb0 con 0x7f626c0a9d70 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f624c005300 con 0x7f626c0a9d70 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f625c001840 con 0x7f626c0a9d70 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.060+0000 7f6268ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235769768 0 0) 0x7f626c13ace0 con 0x7f626c13f650 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c005880 con 0x7f626c13f650 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148751626 0 0) 0x7f624c005880 con 0x7f626c13f650 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f626c13ace0 con 0x7f626c13f650 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6254001b00 con 0x7f626c13f650 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.049 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328303806 0 0) 0x7f624c005300 con 0x7f626c0a9d70 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 >> v1:172.21.15.105:6790/0 conn(0x7f626c13f650 legacy=0x7f626c1394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 >> v1:172.21.15.110:6789/0 conn(0x7f626c1457f0 legacy=0x7f626c134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 monclient: found mon.a 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c1402c0 con 0x7f626c0a9d70 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:58.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 10 monclient: _renew_subs 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c139bd0 con 0x7f626c0a9d70 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 10 monclient: _renew_subs 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f625c001d50 con 0x7f626c0a9d70 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c142e30 con 0x7f626c0a9d70 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f625c001840 con 0x7f626c0a9d70 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:58.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f6268ff9640 10 monclient: dump: 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f626a7fc640 4 set_mon_vals no callback set 2024-03-29T11:11:58.052 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.061+0000 7f62720d9640 1 librados: init done 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.063+0000 7f62720d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.063+0000 7f62720d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.063+0000 7f62720d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.063+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c139e10 con 0x7f626c0a9d70 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.064+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f625c0048e0 con 0x7f626c0a9d70 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f625c0847c0 con 0x7f626c0a9d70 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 10 client.4353.objecter ms_dispatch 0x7f626c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6263fff640 10 monclient: get_auth_request con 0x7f624c06e480 auth_method 0 2024-03-29T11:11:58.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.065+0000 7f6268ff9640 10 client.4353.objecter ms_handle_connect 0x7f624c06e480 2024-03-29T11:11:58.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.073+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f626c139e10 con 0x7f626c0a9d70 2024-03-29T11:11:58.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.073+0000 7f6268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:58.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.073+0000 7f6268ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:58.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.437+0000 7f62720d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:58.415 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.437+0000 7f62720d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:58.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.437+0000 7f62720d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:11:58.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.437+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f626c139e10 con 0x7f626c0a9d70 2024-03-29T11:11:58.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.438+0000 7f6268ff9640 1 -- 172.21.15.105:0/3028040201 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f626c139e10 con 0x7f626c0a9d70 2024-03-29T11:11:58.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.438+0000 7f6268ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:58.416 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.438+0000 7f6268ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:58.417 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:11:58.417 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:58.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.441+0000 7f62720d9640 10 librados: watch_flush enter 2024-03-29T11:11:58.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.441+0000 7f62720d9640 10 librados: watch_flush exit 2024-03-29T11:11:58.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.441+0000 7f62720d9640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:11:58.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.441+0000 7f62720d9640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 >> v1:172.21.15.110:6816/1087970004 conn(0x7f624c06e480 legacy=0x7f624c070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 10 monclient: shutdown 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 >> v1:172.21.15.105:6789/0 conn(0x7f626c0a9d70 legacy=0x7f626c13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 shutdown_connections 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 >> 172.21.15.105:0/3028040201 conn(0x7f626c066a00 msgr2=0x7f626c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 shutdown_connections 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 -- 172.21.15.105:0/3028040201 wait complete. 2024-03-29T11:11:58.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:11:58.442+0000 7f62720d9640 1 librados: shutdown 2024-03-29T11:11:58.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:05.243 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:05.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.400+0000 7ff4001ed640 10 monclient: get_monmap_and_config 2024-03-29T11:12:05.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.400+0000 7ff4001ed640 10 monclient: build_initial_monmap 2024-03-29T11:12:05.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.400+0000 7ff4001ed640 10 monclient: monmap: 2024-03-29T11:12:05.379 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:05.401960+0000 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:05.401960+0000 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.380 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:05.381 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:05.381 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:05.381 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.401+0000 7ff4001ed640 1 Processor -- start 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.402+0000 7ff4001ed640 1 -- start start 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.402+0000 7ff4001ed640 10 monclient: init 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:05.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f80b46b0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f80aeae0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f813fa10 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80b06a0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80b4230 con 0x7ff3f80b46b0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80b0910 con 0x7ff3f813fa10 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff4001ed640 10 monclient(hunting): authenticate will time out at 999.679126s 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fdf89640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff3f80b46b0 0x7ff3f80b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:45738/0 (socket says 172.21.15.105:45738) 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fdf89640 1 -- 172.21.15.105:0/4072038239 learned_addr learned my addr 172.21.15.105:0/4072038239 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092838142 0 0) 0x7ff3f80b4230 con 0x7ff3f80b46b0 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:12:05.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4003520 con 0x7ff3f80b46b0 2024-03-29T11:12:05.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106860646 0 0) 0x7ff3f80b0910 con 0x7ff3f813fa10 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.403+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f80b4230 con 0x7ff3f813fa10 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.404+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1355724029 0 0) 0x7ff3e4003520 con 0x7ff3f80b46b0 2024-03-29T11:12:05.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.404+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 >> v1:172.21.15.105:6790/0 conn(0x7ff3f813fa10 legacy=0x7ff3f814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.404+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 >> v1:172.21.15.105:6789/0 conn(0x7ff3f80aeae0 legacy=0x7ff3f813f300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: _finish_hunting 0 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 1 monclient: found mon.noname-c 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80af6d0 con 0x7ff3f80b46b0 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: _finish_auth 0 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: _check_auth_tickets 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3f4001690 con 0x7ff3f80b46b0 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: dump: 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:05.399 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff4001ed640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff4001ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f40017f0 con 0x7ff3f80b46b0 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 1 -- 172.21.15.105:0/4072038239 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3f4001c00 con 0x7ff3f80b46b0 2024-03-29T11:12:05.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff3fcf87640 10 monclient: dump: 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff4001ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.406+0000 7ff4001ed640 4 set_mon_vals no callback set 2024-03-29T11:12:05.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.407+0000 7ff4001ed640 10 monclient: shutdown 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.407+0000 7ff4001ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.407+0000 7ff4001ed640 1 -- 172.21.15.105:0/4072038239 >> v1:172.21.15.110:6789/0 conn(0x7ff3f80b46b0 legacy=0x7ff3f80b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.408+0000 7ff4001ed640 1 -- 172.21.15.105:0/4072038239 shutdown_connections 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.408+0000 7ff4001ed640 1 -- 172.21.15.105:0/4072038239 >> 172.21.15.105:0/4072038239 conn(0x7ff3f8066a00 msgr2=0x7ff3f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.408+0000 7ff4001ed640 1 -- 172.21.15.105:0/4072038239 shutdown_connections 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.409+0000 7ff4001ed640 1 -- 172.21.15.105:0/4072038239 wait complete. 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.409+0000 7ff4001ed640 10 monclient: build_initial_monmap 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.409+0000 7ff4001ed640 10 monclient: monmap: 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.402 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 librados: starting msgr at 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 librados: starting objecter 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 Processor -- start 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 -- start start 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 librados: setting wanted keys 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 1 librados: calling monclient init 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.410+0000 7ff4001ed640 10 monclient: init 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f80aeae0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:05.403 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f80b46b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f813fa10 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f8137a60 con 0x7ff3f80b46b0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f8144920 con 0x7ff3f80aeae0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f813fee0 con 0x7ff3f813fa10 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff4001ed640 10 monclient(hunting): authenticate will time out at 999.687762s 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff3fdf89640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff3f80aeae0 0x7ff3f81da9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:46100/0 (socket says 172.21.15.105:46100) 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.411+0000 7ff3fdf89640 1 -- 172.21.15.105:0/4276963238 learned_addr learned my addr 172.21.15.105:0/4276963238 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff3f80aeae0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981820226 0 0) 0x7ff3f8144920 con 0x7ff3f80aeae0 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.404 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff3f813fa10 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.412+0000 7ff3ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360747262 0 0) 0x7ff3f813fee0 con 0x7ff3f813fa10 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8144920 con 0x7ff3f813fa10 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165846298 0 0) 0x7ff3dc004eb0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f813fee0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3f4001c00 con 0x7ff3f80aeae0 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.413+0000 7ff3ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.405 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:05.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:05.407 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:05.407 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.407 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:05.408 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:05.409 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.409 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2965903161 0 0) 0x7ff3f8144920 con 0x7ff3f813fa10 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3f813fa10 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3f0001b50 con 0x7ff3f813fa10 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:05.410 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1750777966 0 0) 0x7ff3f813fee0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 >> v1:172.21.15.105:6790/0 conn(0x7ff3f813fa10 legacy=0x7ff3f81dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.414+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 >> v1:172.21.15.105:6789/0 conn(0x7ff3f80b46b0 legacy=0x7ff3f81db760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.415+0000 7ff3ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:05.411 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 1 monclient: found mon.b 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8145c80 con 0x7ff3f80aeae0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 10 monclient: _renew_subs 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f8137540 con 0x7ff3f80aeae0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 10 monclient: _renew_subs 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.412 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f81db2b0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff4001ed640 1 librados: init done 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f4004b40 con 0x7ff3f80aeae0 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3f4004e50 con 0x7ff3f80aeae0 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.416+0000 7ff3ee7fc640 10 monclient: dump: 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:05.413 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:05.414 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:05.414 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:05.414 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:05.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.417+0000 7ff3effff640 4 set_mon_vals no callback set 2024-03-29T11:12:05.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.417+0000 7ff4001ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.418+0000 7ff4001ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.418+0000 7ff4001ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.418+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0005080 con 0x7ff3f80aeae0 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff3f4004fb0 con 0x7ff3f80aeae0 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff3f4084b80 con 0x7ff3f80aeae0 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 10 client.4360.objecter ms_dispatch 0x7ff3f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:05.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3fd788640 10 monclient: get_auth_request con 0x7ff3dc06ddf0 auth_method 0 2024-03-29T11:12:05.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.419+0000 7ff3ee7fc640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:12:05.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.420+0000 7ff3ee7fc640 10 client.4360.objecter ms_handle_connect 0x7ff3dc06ddf0 2024-03-29T11:12:05.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.428+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3f4050290 con 0x7ff3f80aeae0 2024-03-29T11:12:05.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.428+0000 7ff3ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:05.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.428+0000 7ff3ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:05.680 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.702+0000 7ff4001ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.680 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.702+0000 7ff4001ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.680 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.702+0000 7ff4001ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:05.680 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.702+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3e0005330 con 0x7ff3f80aeae0 2024-03-29T11:12:05.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.705+0000 7ff3ee7fc640 1 -- 172.21.15.105:0/4276963238 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff3f4052390 con 0x7ff3f80aeae0 2024-03-29T11:12:05.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.705+0000 7ff3ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:05.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.705+0000 7ff3ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:05.684 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:05.684 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:05.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.707+0000 7ff4001ed640 10 librados: watch_flush enter 2024-03-29T11:12:05.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 10 librados: watch_flush exit 2024-03-29T11:12:05.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:12:05.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:12:05.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff3dc06ddf0 legacy=0x7ff3dc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 10 monclient: shutdown 2024-03-29T11:12:05.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:05.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.708+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 >> v1:172.21.15.110:6789/0 conn(0x7ff3f80aeae0 legacy=0x7ff3f81da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:05.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.710+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 shutdown_connections 2024-03-29T11:12:05.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.710+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 >> 172.21.15.105:0/4276963238 conn(0x7ff3f8066a00 msgr2=0x7ff3f80b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:05.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.710+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 shutdown_connections 2024-03-29T11:12:05.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.710+0000 7ff4001ed640 1 -- 172.21.15.105:0/4276963238 wait complete. 2024-03-29T11:12:05.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:05.710+0000 7ff4001ed640 1 librados: shutdown 2024-03-29T11:12:05.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:12.511 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:12.659 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.681+0000 7f7a35e1e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:12.659 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.681+0000 7f7a35e1e640 10 monclient: build_initial_monmap 2024-03-29T11:12:12.659 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.681+0000 7f7a35e1e640 10 monclient: monmap: 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:12.682277+0000 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:12.682277+0000 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:12.660 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:12.661 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:12.661 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.661 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.682+0000 7f7a35e1e640 1 Processor -- start 2024-03-29T11:12:12.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.683+0000 7f7a35e1e640 1 -- start start 2024-03-29T11:12:12.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.683+0000 7f7a35e1e640 10 monclient: init 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.683+0000 7f7a35e1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.683+0000 7f7a35e1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a300b46b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a300aeae0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a3013fa10 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a300b06a0 con 0x7f7a300aeae0 2024-03-29T11:12:12.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a300b4230 con 0x7f7a3013fa10 2024-03-29T11:12:12.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a300b0910 con 0x7f7a300b46b0 2024-03-29T11:12:12.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:12.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.684+0000 7f7a35e1e640 10 monclient(hunting): authenticate will time out at 1006.960062s 2024-03-29T11:12:12.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2f7fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f7a300b46b0 0x7f7a300b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46498/0 (socket says 172.21.15.105:46498) 2024-03-29T11:12:12.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2f7fe640 1 -- 172.21.15.105:0/4292750210 learned_addr learned my addr 172.21.15.105:0/4292750210 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:12.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879787553 0 0) 0x7f7a300b0910 con 0x7f7a300b46b0 2024-03-29T11:12:12.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:12:12.667 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.685+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a18003520 con 0x7f7a300b46b0 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.686+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328042141 0 0) 0x7f7a18003520 con 0x7f7a300b46b0 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.686+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 >> v1:172.21.15.110:6789/0 conn(0x7f7a3013fa10 legacy=0x7f7a3014d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.686+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 >> v1:172.21.15.105:6789/0 conn(0x7f7a300aeae0 legacy=0x7f7a3013f300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:12.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300af6d0 con 0x7f7a300b46b0 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7a28001a10 con 0x7f7a300b46b0 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: dump: 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.686 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a35e1e640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a35e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a28001be0 con 0x7f7a300b46b0 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a35e1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a35e1e640 4 set_mon_vals no callback set 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 1 -- 172.21.15.105:0/4292750210 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7a28002030 con 0x7f7a300b46b0 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.688+0000 7f7a2e7fc640 10 monclient: dump: 2024-03-29T11:12:12.687 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.689+0000 7f7a35e1e640 10 monclient: shutdown 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.689+0000 7f7a35e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.689+0000 7f7a35e1e640 1 -- 172.21.15.105:0/4292750210 >> v1:172.21.15.105:6790/0 conn(0x7f7a300b46b0 legacy=0x7f7a300b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.689+0000 7f7a35e1e640 1 -- 172.21.15.105:0/4292750210 shutdown_connections 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.689+0000 7f7a35e1e640 1 -- 172.21.15.105:0/4292750210 >> 172.21.15.105:0/4292750210 conn(0x7f7a30066a00 msgr2=0x7f7a30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:12.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.690+0000 7f7a35e1e640 1 -- 172.21.15.105:0/4292750210 shutdown_connections 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.690+0000 7f7a35e1e640 1 -- 172.21.15.105:0/4292750210 wait complete. 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.690+0000 7f7a35e1e640 10 monclient: build_initial_monmap 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.690+0000 7f7a35e1e640 10 monclient: monmap: 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.691+0000 7f7a35e1e640 1 librados: starting msgr at 2024-03-29T11:12:12.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.691+0000 7f7a35e1e640 1 librados: starting objecter 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.691+0000 7f7a35e1e640 1 Processor -- start 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 1 -- start start 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 1 librados: setting wanted keys 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 1 librados: calling monclient init 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 10 monclient: init 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a300aeae0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.692+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a300b46b0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a3013fa10 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a30146040 con 0x7f7a300aeae0 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a30135800 con 0x7f7a300b46b0 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a301464d0 con 0x7f7a3013fa10 2024-03-29T11:12:12.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.693+0000 7f7a35e1e640 10 monclient(hunting): authenticate will time out at 1006.969913s 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.695+0000 7f7a2ffff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f7a3013fa10 0x7f7a301350f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:56852/0 (socket says 172.21.15.105:56852) 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.695+0000 7f7a2ffff640 1 -- 172.21.15.105:0/993388751 learned_addr learned my addr 172.21.15.105:0/993388751 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.695+0000 7f7a17fff640 10 client.?.objecter ms_handle_connect 0x7f7a3013fa10 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.695+0000 7f7a17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745244927 0 0) 0x7f7a301464d0 con 0x7f7a3013fa10 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1c004eb0 con 0x7f7a3013fa10 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 client.?.objecter ms_handle_connect 0x7f7a300aeae0 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.696+0000 7f7a17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761405495 0 0) 0x7f7a30146040 con 0x7f7a300aeae0 2024-03-29T11:12:12.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a301464d0 con 0x7f7a300aeae0 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668912223 0 0) 0x7f7a1c004eb0 con 0x7f7a3013fa10 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a1c005490 con 0x7f7a3013fa10 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7a240017f0 con 0x7f7a3013fa10 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): dump: 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.692 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1756767833 0 0) 0x7f7a1c005490 con 0x7f7a3013fa10 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 >> v1:172.21.15.110:6789/0 conn(0x7f7a300b46b0 legacy=0x7f7a301349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 >> v1:172.21.15.105:6789/0 conn(0x7f7a300aeae0 legacy=0x7f7a301dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 monclient: found mon.c 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30141420 con 0x7f7a3013fa10 2024-03-29T11:12:12.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 10 monclient: _renew_subs 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a30143070 con 0x7f7a3013fa10 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 10 monclient: _renew_subs 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a300b7410 con 0x7f7a3013fa10 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.697+0000 7f7a35e1e640 1 librados: init done 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a24001d00 con 0x7f7a3013fa10 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:12.694 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f7a24004aa0 con 0x7f7a3013fa10 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.698+0000 7f7a17fff640 10 monclient: dump: 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.699+0000 7f7a2d7fa640 4 set_mon_vals no callback set 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.701+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f7a2401aa70 con 0x7f7a3013fa10 2024-03-29T11:12:12.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.701+0000 7f7a35e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.701+0000 7f7a35e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.701+0000 7f7a35e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.701+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a10005080 con 0x7f7a3013fa10 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a2f7fe640 10 monclient: get_auth_request con 0x7f7a1c06df80 auth_method 0 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f7a24084a20 con 0x7f7a3013fa10 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a17fff640 10 client.4403.objecter ms_dispatch 0x7f7a30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a17fff640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a17fff640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.702+0000 7f7a17fff640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:12:12.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.705+0000 7f7a17fff640 10 client.4403.objecter ms_handle_connect 0x7f7a1c06df80 2024-03-29T11:12:12.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.710+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a24050300 con 0x7f7a3013fa10 2024-03-29T11:12:12.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.710+0000 7f7a17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:12.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:12.710+0000 7f7a17fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:13.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.021+0000 7f7a35e1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:13.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.021+0000 7f7a35e1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:13.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.022+0000 7f7a35e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:12:13.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.022+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a10005330 con 0x7f7a3013fa10 2024-03-29T11:12:13.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.023+0000 7f7a17fff640 1 -- 172.21.15.105:0/993388751 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f7a24052230 con 0x7f7a3013fa10 2024-03-29T11:12:13.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.023+0000 7f7a17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:13.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.023+0000 7f7a17fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:13.001 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:13.001 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.026+0000 7f7a35e1e640 10 librados: watch_flush enter 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.026+0000 7f7a35e1e640 10 librados: watch_flush exit 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.026+0000 7f7a35e1e640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.026+0000 7f7a35e1e640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 >> v1:172.21.15.110:6816/1087970004 conn(0x7f7a1c06df80 legacy=0x7f7a1c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 10 monclient: shutdown 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 >> v1:172.21.15.105:6790/0 conn(0x7f7a3013fa10 legacy=0x7f7a301350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 shutdown_connections 2024-03-29T11:12:13.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 >> 172.21.15.105:0/993388751 conn(0x7f7a30066a00 msgr2=0x7f7a3013f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:13.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 shutdown_connections 2024-03-29T11:12:13.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 -- 172.21.15.105:0/993388751 wait complete. 2024-03-29T11:12:13.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:13.027+0000 7f7a35e1e640 1 librados: shutdown 2024-03-29T11:12:13.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:19.829 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:19.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.005+0000 7f8bf8bd1640 10 monclient: get_monmap_and_config 2024-03-29T11:12:19.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.005+0000 7f8bf8bd1640 10 monclient: build_initial_monmap 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.005+0000 7f8bf8bd1640 10 monclient: monmap: 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:20.006538+0000 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:20.006538+0000 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:19.985 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:19.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.007+0000 7f8bf8bd1640 1 Processor -- start 2024-03-29T11:12:19.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.007+0000 7f8bf8bd1640 1 -- start start 2024-03-29T11:12:19.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.007+0000 7f8bf8bd1640 10 monclient: init 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-c con 0x7f8bec034850 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-a con 0x7f8bec0c4ec0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-b con 0x7f8bec0c7d30 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec0c83c0 con 0x7f8bec0c4ec0 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec033280 con 0x7f8bec034850 2024-03-29T11:12:19.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec029100 con 0x7f8bec0c7d30 2024-03-29T11:12:19.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:19.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.008+0000 7f8bf8bd1640 10 monclient(hunting): authenticate will time out at 1014.284688s 2024-03-29T11:12:19.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.014+0000 7f8bf3fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f8bec0c7d30 0x7f8bec029f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41535/0 (socket says 172.21.15.105:41535) 2024-03-29T11:12:19.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.014+0000 7f8bf3fff640 1 -- 172.21.15.105:0/2366404058 learned_addr learned my addr 172.21.15.105:0/2366404058 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:19.994 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394747877 0 0) 0x7f8bec029100 con 0x7f8bec0c7d30 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be4003520 con 0x7f8bec0c7d30 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886253646 0 0) 0x7f8bec033280 con 0x7f8bec034850 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.015+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bec029100 con 0x7f8bec034850 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410921116 0 0) 0x7f8bec029100 con 0x7f8bec034850 2024-03-29T11:12:19.995 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 >> v1:172.21.15.105:6790/0 conn(0x7f8bec0c7d30 legacy=0x7f8bec029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 >> v1:172.21.15.105:6789/0 conn(0x7f8bec0c4ec0 legacy=0x7f8bec029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bec0293e0 con 0x7f8bec034850 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8be8001610 con 0x7f8bec034850 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf27fc640 10 monclient: dump: 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:19.996 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf8bd1640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:12:19.997 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.016+0000 7f8bf8bd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:19.998 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8be8001770 con 0x7f8bec034850 2024-03-29T11:12:19.998 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:19.998 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 1 -- 172.21.15.105:0/2366404058 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8be8001bc0 con 0x7f8bec034850 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.019+0000 7f8bf27fc640 10 monclient: dump: 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.020+0000 7f8bf8bd1640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:19.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.020+0000 7f8bf8bd1640 4 set_mon_vals no callback set 2024-03-29T11:12:20.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.020+0000 7f8bf8bd1640 10 monclient: shutdown 2024-03-29T11:12:20.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.020+0000 7f8bf8bd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:20.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.020+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2366404058 >> v1:172.21.15.110:6789/0 conn(0x7f8bec034850 legacy=0x7f8bec0c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.021+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2366404058 shutdown_connections 2024-03-29T11:12:20.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.021+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2366404058 >> 172.21.15.105:0/2366404058 conn(0x7f8bec0058f0 msgr2=0x7f8bec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.022+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2366404058 shutdown_connections 2024-03-29T11:12:20.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.022+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2366404058 wait complete. 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.023+0000 7f8bf8bd1640 10 monclient: build_initial_monmap 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.023+0000 7f8bf8bd1640 10 monclient: monmap: 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.023+0000 7f8bf8bd1640 1 librados: starting msgr at 2024-03-29T11:12:20.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.023+0000 7f8bf8bd1640 1 librados: starting objecter 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.023+0000 7f8bf8bd1640 1 Processor -- start 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.024+0000 7f8bf8bd1640 1 -- start start 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.024+0000 7f8bf8bd1640 1 librados: setting wanted keys 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.024+0000 7f8bf8bd1640 1 librados: calling monclient init 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.024+0000 7f8bf8bd1640 10 monclient: init 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:20.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-c con 0x7f8bec034850 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-b con 0x7f8bec0c4ec0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient(hunting): picked mon.noname-a con 0x7f8bec0c7d30 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec02daf0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec02dd60 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bec02e040 con 0x7f8bec034850 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:20.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.025+0000 7f8bf8bd1640 10 monclient(hunting): authenticate will time out at 1014.301307s 2024-03-29T11:12:20.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.031+0000 7f8bf2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8bec0c4ec0 0x7f8bec0c9db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:56335/0 (socket says 172.21.15.105:56335) 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.031+0000 7f8bf2ffd640 1 -- 172.21.15.105:0/2528690898 learned_addr learned my addr 172.21.15.105:0/2528690898 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.031+0000 7f8bdffff640 10 client.?.objecter ms_handle_connect 0x7f8bec0c7d30 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.031+0000 7f8bdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.031+0000 7f8bdffff640 10 client.?.objecter ms_handle_connect 0x7f8bec0c4ec0 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.032+0000 7f8bdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.032+0000 7f8bdffff640 10 client.?.objecter ms_handle_connect 0x7f8bec034850 2024-03-29T11:12:20.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.032+0000 7f8bdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529851294 0 0) 0x7f8bec02dd60 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.034+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd8004eb0 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561802510 0 0) 0x7f8bec02e040 con 0x7f8bec034850 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:12:20.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bec02dd60 con 0x7f8bec034850 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885811316 0 0) 0x7f8bec02daf0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bec02e040 con 0x7f8bec0c7d30 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876486754 0 0) 0x7f8bec02dd60 con 0x7f8bec034850 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bec02daf0 con 0x7f8bec034850 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8be8001a80 con 0x7f8bec034850 2024-03-29T11:12:20.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): dump: 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527944685 0 0) 0x7f8bec02e040 con 0x7f8bec0c7d30 2024-03-29T11:12:20.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bec02dd60 con 0x7f8bec0c7d30 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8bf404fdd0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.035+0000 7f8bdffff640 10 monclient(hunting): dump: 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:20.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.036+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810812664 0 0) 0x7f8bd8004eb0 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.036+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bec02e040 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.039+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8be0001960 con 0x7f8bec0c4ec0 2024-03-29T11:12:20.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.039+0000 7f8bdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:20.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.039+0000 7f8bdffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:20.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 10 monclient(hunting): dump: 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2268596249 0 0) 0x7f8bec02dd60 con 0x7f8bec0c7d30 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 >> v1:172.21.15.105:6790/0 conn(0x7f8bec034850 legacy=0x7f8bec02c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 >> v1:172.21.15.110:6789/0 conn(0x7f8bec0c4ec0 legacy=0x7f8bec0c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 1 monclient: found mon.a 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bec0cba60 con 0x7f8bec0c7d30 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bf8bd1640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bf8bd1640 10 monclient: _renew_subs 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bf8bd1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.040+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bec0caf10 con 0x7f8bec0c7d30 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf8bd1640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf8bd1640 10 monclient: _renew_subs 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf8bd1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:20.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bf404f400 con 0x7f8bec0c7d30 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf8bd1640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bec0b1ef0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf8bd1640 1 librados: init done 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.041+0000 7f8bf1ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.042+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8bf4075f10 con 0x7f8bec0c7d30 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.042+0000 7f8bdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.042+0000 7f8bdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.042+0000 7f8bdffff640 10 monclient: dump: 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:20.025 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.043+0000 7f8bddffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.043+0000 7f8bddffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.043+0000 7f8bddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:20.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.043+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bec0cb8b0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.046+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8bf4067bf0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8bf40cf090 con 0x7f8bec0c7d30 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 10 client.4359.objecter ms_dispatch 0x7f8bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bf37fe640 10 monclient: get_auth_request con 0x7f8bd806dd20 auth_method 0 2024-03-29T11:12:20.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.047+0000 7f8bdffff640 10 client.4359.objecter ms_handle_connect 0x7f8bd806dd20 2024-03-29T11:12:20.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.056+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bf40cb330 con 0x7f8bec0c7d30 2024-03-29T11:12:20.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.056+0000 7f8bdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:20.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.056+0000 7f8bdffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:20.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.382+0000 7f8bddffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.382+0000 7f8bddffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.382+0000 7f8bddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:20.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.382+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bec0cbbf0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.383+0000 7f8bdffff640 1 -- 172.21.15.105:0/2528690898 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8bf40cb4d0 con 0x7f8bec0c7d30 2024-03-29T11:12:20.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.383+0000 7f8bdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:20.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.383+0000 7f8bdffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:20.362 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:20.362 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:20.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.387+0000 7f8bddffb640 10 librados: watch_flush enter 2024-03-29T11:12:20.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.387+0000 7f8bddffb640 10 librados: watch_flush exit 2024-03-29T11:12:20.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.387+0000 7f8bddffb640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.387+0000 7f8bddffb640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.387+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 >> v1:172.21.15.110:6816/1087970004 conn(0x7f8bd806dd20 legacy=0x7f8bd80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.388+0000 7f8bddffb640 10 monclient: shutdown 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.388+0000 7f8bddffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.388+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 >> v1:172.21.15.105:6789/0 conn(0x7f8bec0c7d30 legacy=0x7f8bec16b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.388+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 shutdown_connections 2024-03-29T11:12:20.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.388+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 >> 172.21.15.105:0/2528690898 conn(0x7f8bec0058f0 msgr2=0x7f8bec0c1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:20.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.389+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 shutdown_connections 2024-03-29T11:12:20.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.389+0000 7f8bddffb640 1 -- 172.21.15.105:0/2528690898 wait complete. 2024-03-29T11:12:20.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:20.389+0000 7f8bddffb640 1 librados: shutdown 2024-03-29T11:12:20.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:27.193 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.369+0000 7fb13f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.369+0000 7fb13f59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.369+0000 7fb13f59e640 10 monclient: monmap: 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:27.370291+0000 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:27.370291+0000 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:27.348 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:27.349 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:27.349 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.370+0000 7fb13f59e640 1 Processor -- start 2024-03-29T11:12:27.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.370+0000 7fb13f59e640 1 -- start start 2024-03-29T11:12:27.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.370+0000 7fb13f59e640 10 monclient: init 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.371+0000 7fb13f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1400b4820 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1400a9ce0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1400b3cd0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1400b4180 con 0x7fb1400a9ce0 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1400aaf60 con 0x7fb1400b4820 2024-03-29T11:12:27.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14013c200 con 0x7fb1400b3cd0 2024-03-29T11:12:27.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:27.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.372+0000 7fb13f59e640 10 monclient(hunting): authenticate will time out at 1021.648089s 2024-03-29T11:12:27.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.375+0000 7fb13e59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb1400b4820 0x7fb14013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:46847/0 (socket says 172.21.15.105:46847) 2024-03-29T11:12:27.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.375+0000 7fb13e59c640 1 -- 172.21.15.105:0/2410212471 learned_addr learned my addr 172.21.15.105:0/2410212471 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680754634 0 0) 0x7fb1400aaf60 con 0x7fb1400b4820 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:12:27.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.376+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb128003520 con 0x7fb1400b4820 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662423422 0 0) 0x7fb128003520 con 0x7fb1400b4820 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 >> v1:172.21.15.105:6790/0 conn(0x7fb1400b3cd0 legacy=0x7fb14014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 >> v1:172.21.15.105:6789/0 conn(0x7fb1400a9ce0 legacy=0x7fb140145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 monclient: found mon.noname-c 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb140145e80 con 0x7fb1400b4820 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: _finish_auth 0 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13f59e640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:12:27.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:27.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb130001670 con 0x7fb1400b4820 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: dump: 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.360 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:27.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1300018d0 con 0x7fb1400b4820 2024-03-29T11:12:27.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:27.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:27.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13f59e640 4 set_mon_vals no callback set 2024-03-29T11:12:27.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 1 -- 172.21.15.105:0/2410212471 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb130001670 con 0x7fb1400b4820 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.377+0000 7fb13d59a640 10 monclient: dump: 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.378+0000 7fb13f59e640 10 monclient: shutdown 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.378+0000 7fb13f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.378+0000 7fb13f59e640 1 -- 172.21.15.105:0/2410212471 >> v1:172.21.15.110:6789/0 conn(0x7fb1400b4820 legacy=0x7fb14013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.378+0000 7fb13f59e640 1 -- 172.21.15.105:0/2410212471 shutdown_connections 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.378+0000 7fb13f59e640 1 -- 172.21.15.105:0/2410212471 >> 172.21.15.105:0/2410212471 conn(0x7fb14006d070 msgr2=0x7fb140066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.380+0000 7fb13f59e640 1 -- 172.21.15.105:0/2410212471 shutdown_connections 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.382+0000 7fb13f59e640 1 -- 172.21.15.105:0/2410212471 wait complete. 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.383+0000 7fb13f59e640 10 monclient: build_initial_monmap 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.383+0000 7fb13f59e640 10 monclient: monmap: 2024-03-29T11:12:27.364 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.383+0000 7fb13f59e640 1 librados: starting msgr at 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.383+0000 7fb13f59e640 1 librados: starting objecter 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 Processor -- start 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 -- start start 2024-03-29T11:12:27.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 librados: setting wanted keys 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 librados: calling monclient init 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient: init 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1400b3cd0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1400b4820 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1400b12d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14006ed40 con 0x7fb1400b12d0 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14006ea10 con 0x7fb1400b4820 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb140070b00 con 0x7fb1400b3cd0 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:27.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.384+0000 7fb13f59e640 10 monclient(hunting): authenticate will time out at 1021.660968s 2024-03-29T11:12:27.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.390+0000 7fb13ed9d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fb1400b12d0 0x7fb1400b0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:45063/0 (socket says 172.21.15.105:45063) 2024-03-29T11:12:27.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.390+0000 7fb13ed9d640 1 -- 172.21.15.105:0/3524403594 learned_addr learned my addr 172.21.15.105:0/3524403594 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb1400b12d0 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283960428 0 0) 0x7fb14006ed40 con 0x7fb1400b12d0 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.391+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:12:27.370 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120004eb0 con 0x7fb1400b12d0 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb1400b4820 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb1400b3cd0 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248683082 0 0) 0x7fb14006ea10 con 0x7fb1400b4820 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.392+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb14006ed40 con 0x7fb1400b4820 2024-03-29T11:12:27.374 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194113314 0 0) 0x7fb140070b00 con 0x7fb1400b3cd0 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb14006ea10 con 0x7fb1400b3cd0 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776485266 0 0) 0x7fb120004eb0 con 0x7fb1400b12d0 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb140070b00 con 0x7fb1400b12d0 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb138004380 con 0x7fb1400b12d0 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): dump: 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:27.375 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.393+0000 7fb12effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3255356407 0 0) 0x7fb140070b00 con 0x7fb1400b12d0 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 >> v1:172.21.15.105:6790/0 conn(0x7fb1400b3cd0 legacy=0x7fb1401d5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 >> v1:172.21.15.110:6789/0 conn(0x7fb1400b4820 legacy=0x7fb14006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 monclient: found mon.a 2024-03-29T11:12:27.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1401d65b0 con 0x7fb1400b12d0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: _finish_auth 0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 10 monclient: _renew_subs 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb140070f90 con 0x7fb1400b12d0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 10 monclient: _renew_subs 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1400b0a80 con 0x7fb1400b12d0 2024-03-29T11:12:27.377 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13f59e640 1 librados: init done 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb138001250 con 0x7fb1400b12d0 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb138001620 con 0x7fb1400b12d0 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb13cd99640 4 set_mon_vals no callback set 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.394+0000 7fb12effd640 10 monclient: dump: 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:27.378 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:27.379 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:27.379 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.397+0000 7fb13f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.397+0000 7fb13f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.397+0000 7fb13f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.397+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124005080 con 0x7fb1400b12d0 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.399+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb138001780 con 0x7fb1400b12d0 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fb138084d80 con 0x7fb1400b12d0 2024-03-29T11:12:27.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb12effd640 10 client.4362.objecter ms_dispatch 0x7fb140000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:27.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb12effd640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:27.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb12effd640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:27.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb12effd640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:12:27.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.400+0000 7fb13e59c640 10 monclient: get_auth_request con 0x7fb12006de30 auth_method 0 2024-03-29T11:12:27.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.401+0000 7fb12effd640 10 client.4362.objecter ms_handle_connect 0x7fb12006de30 2024-03-29T11:12:27.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.412+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb138050660 con 0x7fb1400b12d0 2024-03-29T11:12:27.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.412+0000 7fb12effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:27.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.412+0000 7fb12effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:27.718 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.740+0000 7fb13f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.740+0000 7fb13f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.740+0000 7fb13f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:27.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.740+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb124005330 con 0x7fb1400b12d0 2024-03-29T11:12:27.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.741+0000 7fb12effd640 1 -- 172.21.15.105:0/3524403594 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fb138052590 con 0x7fb1400b12d0 2024-03-29T11:12:27.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.741+0000 7fb12effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:27.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.741+0000 7fb12effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:27.720 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:27.720 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 10 librados: watch_flush enter 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 10 librados: watch_flush exit 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb12006de30 legacy=0x7fb1200702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 10 monclient: shutdown 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 >> v1:172.21.15.105:6789/0 conn(0x7fb1400b12d0 legacy=0x7fb1400b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 shutdown_connections 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.745+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 >> 172.21.15.105:0/3524403594 conn(0x7fb14006d070 msgr2=0x7fb1400b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.746+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 shutdown_connections 2024-03-29T11:12:27.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.746+0000 7fb13f59e640 1 -- 172.21.15.105:0/3524403594 wait complete. 2024-03-29T11:12:27.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:27.746+0000 7fb13f59e640 1 librados: shutdown 2024-03-29T11:12:27.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:34.549 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:34.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.727+0000 7f69d09e6640 10 monclient: get_monmap_and_config 2024-03-29T11:12:34.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.727+0000 7f69d09e6640 10 monclient: build_initial_monmap 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.727+0000 7f69d09e6640 10 monclient: monmap: 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:34.728894+0000 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:34.728894+0000 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:34.707 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.731+0000 7f69d09e6640 1 Processor -- start 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.731+0000 7f69d09e6640 1 -- start start 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.731+0000 7f69d09e6640 10 monclient: init 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-b con 0x7f69cc0b0860 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-c con 0x7f69cc13ef30 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-a con 0x7f69cc0af8e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0b0c40 con 0x7f69cc0af8e0 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0aed10 con 0x7f69cc13ef30 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0afd50 con 0x7f69cc0b0860 2024-03-29T11:12:34.712 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:34.713 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.732+0000 7f69d09e6640 10 monclient(hunting): authenticate will time out at 1029.008423s 2024-03-29T11:12:34.714 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.736+0000 7f69c9d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f69cc13ef30 0x7f69cc0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:36163/0 (socket says 172.21.15.105:36163) 2024-03-29T11:12:34.714 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.736+0000 7f69c9d9b640 1 -- 172.21.15.105:0/2638064334 learned_addr learned my addr 172.21.15.105:0/2638064334 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:34.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895645404 0 0) 0x7f69cc0aed10 con 0x7f69cc13ef30 2024-03-29T11:12:34.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:12:34.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.738+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b4003520 con 0x7f69cc13ef30 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087669847 0 0) 0x7f69cc0afd50 con 0x7f69cc0b0860 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc0aed10 con 0x7f69cc0b0860 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2814113321 0 0) 0x7f69b4003520 con 0x7f69cc13ef30 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 >> v1:172.21.15.105:6790/0 conn(0x7f69cc0b0860 legacy=0x7f69cc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 >> v1:172.21.15.105:6789/0 conn(0x7f69cc0af8e0 legacy=0x7f69cc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.719 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 monclient: found mon.noname-c 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0ab1b0 con 0x7f69cc13ef30 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: _finish_auth 0 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f69c4005010 con 0x7f69cc13ef30 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: dump: 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:34.720 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69d09e6640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69d09e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69c4005230 con 0x7f69cc13ef30 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 1 -- 172.21.15.105:0/2638064334 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f69c4001790 con 0x7f69cc13ef30 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.721 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69c959a640 10 monclient: dump: 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69d09e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.739+0000 7f69d09e6640 4 set_mon_vals no callback set 2024-03-29T11:12:34.722 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.740+0000 7f69d09e6640 10 monclient: shutdown 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.740+0000 7f69d09e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.740+0000 7f69d09e6640 1 -- 172.21.15.105:0/2638064334 >> v1:172.21.15.110:6789/0 conn(0x7f69cc13ef30 legacy=0x7f69cc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.741+0000 7f69d09e6640 1 -- 172.21.15.105:0/2638064334 shutdown_connections 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.741+0000 7f69d09e6640 1 -- 172.21.15.105:0/2638064334 >> 172.21.15.105:0/2638064334 conn(0x7f69cc06b660 msgr2=0x7f69cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.741+0000 7f69d09e6640 1 -- 172.21.15.105:0/2638064334 shutdown_connections 2024-03-29T11:12:34.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.741+0000 7f69d09e6640 1 -- 172.21.15.105:0/2638064334 wait complete. 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.742+0000 7f69d09e6640 10 monclient: build_initial_monmap 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.742+0000 7f69d09e6640 10 monclient: monmap: 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:34.724 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.742+0000 7f69d09e6640 1 librados: starting msgr at 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.742+0000 7f69d09e6640 1 librados: starting objecter 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.743+0000 7f69d09e6640 1 Processor -- start 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.743+0000 7f69d09e6640 1 -- start start 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.743+0000 7f69d09e6640 1 librados: setting wanted keys 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.743+0000 7f69d09e6640 1 librados: calling monclient init 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.743+0000 7f69d09e6640 10 monclient: init 2024-03-29T11:12:34.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-c con 0x7f69cc0af8e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-b con 0x7f69cc0b0860 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient(hunting): picked mon.noname-a con 0x7f69cc13ef30 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0b13e0 con 0x7f69cc13ef30 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0b1930 con 0x7f69cc0b0860 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc137850 con 0x7f69cc0af8e0 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:34.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.744+0000 7f69d09e6640 10 monclient(hunting): authenticate will time out at 1029.020448s 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.748+0000 7f69c9d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f69cc0b0860 0x7f69cc135d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:36229/0 (socket says 172.21.15.105:36229) 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.748+0000 7f69c9d9b640 1 -- 172.21.15.105:0/4238296491 learned_addr learned my addr 172.21.15.105:0/4238296491 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.750+0000 7f69baffd640 10 client.?.objecter ms_handle_connect 0x7f69cc0b0860 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.750+0000 7f69baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532638748 0 0) 0x7f69cc0b1930 con 0x7f69cc0b0860 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ac004eb0 con 0x7f69cc0b0860 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 client.?.objecter ms_handle_connect 0x7f69cc13ef30 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.751+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533317438 0 0) 0x7f69cc0b13e0 con 0x7f69cc13ef30 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc0b1930 con 0x7f69cc13ef30 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198290384 0 0) 0x7f69cc0b1930 con 0x7f69cc13ef30 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69cc0b13e0 con 0x7f69cc13ef30 2024-03-29T11:12:34.732 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f69bc0017e0 con 0x7f69cc13ef30 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): dump: 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:34.733 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 client.?.objecter ms_handle_connect 0x7f69cc0af8e0 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3092414565 0 0) 0x7f69cc0b13e0 con 0x7f69cc13ef30 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 >> v1:172.21.15.105:6790/0 conn(0x7f69cc0af8e0 legacy=0x7f69cc135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 >> v1:172.21.15.110:6789/0 conn(0x7f69cc0b0860 legacy=0x7f69cc135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 monclient: found mon.a 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0b1650 con 0x7f69cc13ef30 2024-03-29T11:12:34.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69bc001c90 con 0x7f69cc13ef30 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69c8d99640 4 set_mon_vals no callback set 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f69bc0037b0 con 0x7f69cc13ef30 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.752+0000 7f69baffd640 10 monclient: dump: 2024-03-29T11:12:34.735 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 10 monclient: _renew_subs 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:34.736 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc136ba0 con 0x7f69cc13ef30 2024-03-29T11:12:34.737 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:34.737 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 10 monclient: _renew_subs 2024-03-29T11:12:34.737 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:34.737 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc13aff0 con 0x7f69cc13ef30 2024-03-29T11:12:34.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.753+0000 7f69d09e6640 1 librados: init done 2024-03-29T11:12:34.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69d09e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69d09e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69d09e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:34.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f69bc003910 con 0x7f69cc13ef30 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc136ea0 con 0x7f69cc13ef30 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f69bc0837c0 con 0x7f69cc13ef30 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69baffd640 10 client.4368.objecter ms_dispatch 0x7f69cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69baffd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.755+0000 7f69baffd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.756+0000 7f69baffd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.758+0000 7f69ca59c640 10 monclient: get_auth_request con 0x7f69ac06dd90 auth_method 0 2024-03-29T11:12:34.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.760+0000 7f69baffd640 10 client.4368.objecter ms_handle_connect 0x7f69ac06dd90 2024-03-29T11:12:34.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.765+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69cc136ea0 con 0x7f69cc13ef30 2024-03-29T11:12:34.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.765+0000 7f69baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:34.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:34.765+0000 7f69baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69d09e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69d09e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69d09e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69cc1376b0 con 0x7f69cc13ef30 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69baffd640 1 -- 172.21.15.105:0/4238296491 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f69cc1376b0 con 0x7f69cc13ef30 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:35.071 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.093+0000 7f69baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:35.073 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:35.074 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:35.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 10 librados: watch_flush enter 2024-03-29T11:12:35.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 10 librados: watch_flush exit 2024-03-29T11:12:35.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:12:35.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:12:35.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 >> v1:172.21.15.110:6816/1087970004 conn(0x7f69ac06dd90 legacy=0x7f69ac070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 10 monclient: shutdown 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 >> v1:172.21.15.105:6789/0 conn(0x7f69cc13ef30 legacy=0x7f69cc1e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 shutdown_connections 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.099+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 >> 172.21.15.105:0/4238296491 conn(0x7f69cc06b660 msgr2=0x7f69cc13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.100+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 shutdown_connections 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.100+0000 7f69d09e6640 1 -- 172.21.15.105:0/4238296491 wait complete. 2024-03-29T11:12:35.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:35.100+0000 7f69d09e6640 1 librados: shutdown 2024-03-29T11:12:35.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:41.906 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:42.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.071+0000 7f8a4498c640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.071+0000 7f8a4498c640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.071+0000 7f8a4498c640 10 monclient: monmap: 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:42.072299+0000 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:42.072299+0000 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:42.050 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.072+0000 7f8a4498c640 1 Processor -- start 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.072+0000 7f8a4498c640 1 -- start start 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.072+0000 7f8a4498c640 10 monclient: init 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a400b0860 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4013ef30 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:42.053 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a400af8e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400b0c40 con 0x7f8a400b0860 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400aed10 con 0x7f8a400af8e0 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400afd50 con 0x7f8a4013ef30 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.073+0000 7f8a4498c640 10 monclient(hunting): authenticate will time out at 1036.349863s 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.075+0000 7f8a3e59c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f8a400b0860 0x7f8a4013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:49896/0 (socket says 172.21.15.105:49896) 2024-03-29T11:12:42.054 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.075+0000 7f8a3e59c640 1 -- 172.21.15.105:0/1135884883 learned_addr learned my addr 172.21.15.105:0/1135884883 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393917314 0 0) 0x7f8a400b0c40 con 0x7f8a400b0860 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:12:42.055 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.076+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c003520 con 0x7f8a400b0860 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.079+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011304470 0 0) 0x7f8a400afd50 con 0x7f8a4013ef30 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a400b0c40 con 0x7f8a4013ef30 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638306945 0 0) 0x7f8a2c003520 con 0x7f8a400b0860 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 >> v1:172.21.15.105:6790/0 conn(0x7f8a4013ef30 legacy=0x7f8a400af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.058 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 >> v1:172.21.15.110:6789/0 conn(0x7f8a400af8e0 legacy=0x7f8a40149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 monclient: found mon.noname-a 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400ab1b0 con 0x7f8a400b0860 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a38001690 con 0x7f8a400b0860 2024-03-29T11:12:42.059 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a3d59a640 10 monclient: dump: 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:42.060 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:42.061 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a4498c640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:12:42.061 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.080+0000 7f8a4498c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a380017f0 con 0x7f8a400b0860 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 1 -- 172.21.15.105:0/1135884883 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a38001bc0 con 0x7f8a400b0860 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.081+0000 7f8a3d59a640 10 monclient: dump: 2024-03-29T11:12:42.062 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:42.063 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.064 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.082+0000 7f8a4498c640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.064 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.082+0000 7f8a4498c640 4 set_mon_vals no callback set 2024-03-29T11:12:42.064 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.083+0000 7f8a4498c640 10 monclient: shutdown 2024-03-29T11:12:42.065 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.083+0000 7f8a4498c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.065 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.083+0000 7f8a4498c640 1 -- 172.21.15.105:0/1135884883 >> v1:172.21.15.105:6789/0 conn(0x7f8a400b0860 legacy=0x7f8a4013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.065 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.084+0000 7f8a4498c640 1 -- 172.21.15.105:0/1135884883 shutdown_connections 2024-03-29T11:12:42.065 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.084+0000 7f8a4498c640 1 -- 172.21.15.105:0/1135884883 >> 172.21.15.105:0/1135884883 conn(0x7f8a4006b660 msgr2=0x7f8a4006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.065 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.087+0000 7f8a4498c640 1 -- 172.21.15.105:0/1135884883 shutdown_connections 2024-03-29T11:12:42.066 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.087+0000 7f8a4498c640 1 -- 172.21.15.105:0/1135884883 wait complete. 2024-03-29T11:12:42.067 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.088+0000 7f8a4498c640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.088+0000 7f8a4498c640 10 monclient: monmap: 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:42.068 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:42.069 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:42.069 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.088+0000 7f8a4498c640 1 librados: starting msgr at 2024-03-29T11:12:42.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.088+0000 7f8a4498c640 1 librados: starting objecter 2024-03-29T11:12:42.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.088+0000 7f8a4498c640 1 Processor -- start 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.089+0000 7f8a4498c640 1 -- start start 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.089+0000 7f8a4498c640 1 librados: setting wanted keys 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.089+0000 7f8a4498c640 1 librados: calling monclient init 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.089+0000 7f8a4498c640 10 monclient: init 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a400af8e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4013ef30 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a40137c60 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:42.076 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a40138240 con 0x7f8a4013ef30 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4013b480 con 0x7f8a40137c60 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4013b610 con 0x7f8a400af8e0 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.090+0000 7f8a4498c640 10 monclient(hunting): authenticate will time out at 1036.366521s 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a3dd9b640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f8a4013ef30 0x7f8a401db700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:52714/0 (socket says 172.21.15.105:52714) 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a3dd9b640 1 -- 172.21.15.105:0/3091580595 learned_addr learned my addr 172.21.15.105:0/3091580595 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a36ffd640 10 client.?.objecter ms_handle_connect 0x7f8a4013ef30 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449329785 0 0) 0x7f8a40138240 con 0x7f8a4013ef30 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.091+0000 7f8a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a20004eb0 con 0x7f8a4013ef30 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4176754012 0 0) 0x7f8a20004eb0 con 0x7f8a4013ef30 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a40138240 con 0x7f8a4013ef30 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.092+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a30001d30 con 0x7f8a4013ef30 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.078 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096419437 0 0) 0x7f8a40138240 con 0x7f8a4013ef30 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.093+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 >> v1:172.21.15.105:6790/0 conn(0x7f8a400af8e0 legacy=0x7f8a400b3930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 >> v1:172.21.15.110:6789/0 conn(0x7f8a40137c60 legacy=0x7f8a401dbe10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 1 monclient: found mon.a 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.095+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4013b7a0 con 0x7f8a4013ef30 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a36ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a36ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 10 monclient: _renew_subs 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a400b4040 con 0x7f8a4013ef30 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 10 monclient: _renew_subs 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a40137670 con 0x7f8a4013ef30 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.096+0000 7f8a4498c640 1 librados: init done 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a30001e90 con 0x7f8a4013ef30 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a30003de0 con 0x7f8a4013ef30 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a37fff640 4 set_mon_vals no callback set 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.097+0000 7f8a36ffd640 10 monclient: dump: 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:42.081 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.099+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8a30003f40 con 0x7f8a4013ef30 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.100+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8a30084f50 con 0x7f8a4013ef30 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.100+0000 7f8a36ffd640 10 client.4374.objecter ms_dispatch 0x7f8a40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.100+0000 7f8a36ffd640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.100+0000 7f8a36ffd640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.100+0000 7f8a36ffd640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.101+0000 7f8a3e59c640 10 monclient: get_auth_request con 0x7f8a2006dc80 auth_method 0 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.101+0000 7f8a4498c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.101+0000 7f8a4498c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.101+0000 7f8a4498c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.101+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24005080 con 0x7f8a4013ef30 2024-03-29T11:12:42.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.104+0000 7f8a36ffd640 10 client.4374.objecter ms_handle_connect 0x7f8a2006dc80 2024-03-29T11:12:42.089 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.109+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a30050830 con 0x7f8a4013ef30 2024-03-29T11:12:42.089 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.109+0000 7f8a36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.089 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.109+0000 7f8a36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.466+0000 7f8a4498c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.466+0000 7f8a4498c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.466+0000 7f8a4498c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:42.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.466+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a24005330 con 0x7f8a4013ef30 2024-03-29T11:12:42.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.475+0000 7f8a36ffd640 1 -- 172.21.15.105:0/3091580595 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8a30052760 con 0x7f8a4013ef30 2024-03-29T11:12:42.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.475+0000 7f8a36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:42.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.475+0000 7f8a36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.455 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:42.455 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:42.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.483+0000 7f8a4498c640 10 librados: watch_flush enter 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 10 librados: watch_flush exit 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 >> v1:172.21.15.110:6816/1087970004 conn(0x7f8a2006dc80 legacy=0x7f8a20070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 10 monclient: shutdown 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 >> v1:172.21.15.105:6789/0 conn(0x7f8a4013ef30 legacy=0x7f8a401db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.484+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 shutdown_connections 2024-03-29T11:12:42.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.485+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 >> 172.21.15.105:0/3091580595 conn(0x7f8a4006b660 msgr2=0x7f8a401c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.485+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 shutdown_connections 2024-03-29T11:12:42.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.485+0000 7f8a4498c640 1 -- 172.21.15.105:0/3091580595 wait complete. 2024-03-29T11:12:42.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:42.487+0000 7f8a4498c640 1 librados: shutdown 2024-03-29T11:12:42.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:49.293 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:49.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.447+0000 7fcdc243e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.447+0000 7fcdc243e640 10 monclient: build_initial_monmap 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.447+0000 7fcdc243e640 10 monclient: monmap: 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:49.448241+0000 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:49.448241+0000 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.426 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:49.427 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:49.427 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:49.427 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.448+0000 7fcdc243e640 1 Processor -- start 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.448+0000 7fcdc243e640 1 -- start start 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.448+0000 7fcdc243e640 10 monclient: init 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdbc0acf60 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdbc13ff40 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-b con 0x7fcdbc1403f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:49.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0b4310 con 0x7fcdbc0acf60 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc140800 con 0x7fcdbc13ff40 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0aec00 con 0x7fcdbc1403f0 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc243e640 10 monclient(hunting): authenticate will time out at 1043.725541s 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdbbfff640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fcdbc0acf60 0x7fcdbc139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:58022/0 (socket says 172.21.15.105:58022) 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdc09db640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fcdbc1403f0 0x7fcdbc14bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:39176/0 (socket says 172.21.15.105:39176) 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.449+0000 7fcdbbfff640 1 -- 172.21.15.105:0/3926419151 learned_addr learned my addr 172.21.15.105:0/3926419151 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096787165 0 0) 0x7fcdbc0b4310 con 0x7fcdbc0acf60 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda4003520 con 0x7fcdbc0acf60 2024-03-29T11:12:49.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821145495 0 0) 0x7fcdbc0aec00 con 0x7fcdbc1403f0 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdbc0b4310 con 0x7fcdbc1403f0 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981062655 0 0) 0x7fcdbc0b4310 con 0x7fcdbc1403f0 2024-03-29T11:12:49.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.450+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 >> v1:172.21.15.110:6789/0 conn(0x7fcdbc13ff40 legacy=0x7fcdbc148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 >> v1:172.21.15.105:6789/0 conn(0x7fcdbc0acf60 legacy=0x7fcdbc139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 1 monclient: found mon.noname-b 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0aeee0 con 0x7fcdbc1403f0 2024-03-29T11:12:49.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcdac0015f0 con 0x7fcdbc1403f0 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.452+0000 7fcdbaffd640 10 monclient: dump: 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:49.435 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdac001750 con 0x7fcdbc1403f0 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 1 -- 172.21.15.105:0/3926419151 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcdac001b60 con 0x7fcdbc1403f0 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdbaffd640 10 monclient: dump: 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:49.436 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdc243e640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdc243e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.453+0000 7fcdc243e640 4 set_mon_vals no callback set 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.454+0000 7fcdc243e640 10 monclient: shutdown 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.454+0000 7fcdc243e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.454+0000 7fcdc243e640 1 -- 172.21.15.105:0/3926419151 >> v1:172.21.15.105:6790/0 conn(0x7fcdbc1403f0 legacy=0x7fcdbc14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.455+0000 7fcdc243e640 1 -- 172.21.15.105:0/3926419151 shutdown_connections 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.455+0000 7fcdc243e640 1 -- 172.21.15.105:0/3926419151 >> 172.21.15.105:0/3926419151 conn(0x7fcdbc0b35e0 msgr2=0x7fcdbc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:49.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.455+0000 7fcdc243e640 1 -- 172.21.15.105:0/3926419151 shutdown_connections 2024-03-29T11:12:49.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.456+0000 7fcdc243e640 1 -- 172.21.15.105:0/3926419151 wait complete. 2024-03-29T11:12:49.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.457+0000 7fcdc243e640 10 monclient: build_initial_monmap 2024-03-29T11:12:49.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.457+0000 7fcdc243e640 10 monclient: monmap: 2024-03-29T11:12:49.445 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 librados: starting msgr at 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 librados: starting objecter 2024-03-29T11:12:49.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 Processor -- start 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 -- start start 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 librados: setting wanted keys 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 1 librados: calling monclient init 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.458+0000 7fcdc243e640 10 monclient: init 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.459+0000 7fcdc243e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.459+0000 7fcdc243e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.459+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdbc0acf60 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdbc13ff40 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdbbfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fcdbc0acf60 0x7fcdbc1df0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41216/0 (socket says 172.21.15.105:41216) 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 10 monclient(hunting): picked mon.noname-b con 0x7fcdbc1403f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc1de930 con 0x7fcdbc13ff40 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc1df800 con 0x7fcdbc1403f0 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc1ca3e0 con 0x7fcdbc0acf60 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:49.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdc243e640 10 monclient(hunting): authenticate will time out at 1043.736887s 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.460+0000 7fcdbbfff640 1 -- 172.21.15.105:0/708721121 learned_addr learned my addr 172.21.15.105:0/708721121 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 client.?.objecter ms_handle_connect 0x7fcdbc0acf60 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 client.?.objecter ms_handle_connect 0x7fcdbc13ff40 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499196831 0 0) 0x7fcdbc1de930 con 0x7fcdbc13ff40 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda8004eb0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385650751 0 0) 0x7fcdbc1ca3e0 con 0x7fcdbc0acf60 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:49.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdbc1de930 con 0x7fcdbc0acf60 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966796728 0 0) 0x7fcda8004eb0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdbc1ca3e0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcdb0001870 con 0x7fcdbc13ff40 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.449 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.450 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:49.450 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:49.450 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:49.454 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.461+0000 7fcdb8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:49.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3360889276 0 0) 0x7fcdbc1ca3e0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 >> v1:172.21.15.105:6790/0 conn(0x7fcdbc0acf60 legacy=0x7fcdbc1df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 >> v1:172.21.15.110:6789/0 conn(0x7fcdbc1403f0 legacy=0x7fcdbc1c8b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 1 monclient: found mon.a 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc1debd0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.462+0000 7fcdb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdb0001d80 con 0x7fcdbc13ff40 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcdb0003830 con 0x7fcdbc13ff40 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:49.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdb8ff9640 10 monclient: dump: 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:49.456 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdba7fc640 4 set_mon_vals no callback set 2024-03-29T11:12:49.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:12:49.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 10 monclient: _renew_subs 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc1cb0c0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 10 monclient: _renew_subs 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc0acac0 con 0x7fcdbc13ff40 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.463+0000 7fcdc243e640 1 librados: init done 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.464+0000 7fcdc243e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.464+0000 7fcdc243e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.465+0000 7fcdc243e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.465+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc0ad340 con 0x7fcdbc13ff40 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.465+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fcdb0019800 con 0x7fcdbc13ff40 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fcdb0083850 con 0x7fcdbc13ff40 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 10 client.4380.objecter ms_dispatch 0x7fcdbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:12:49.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdbbfff640 10 monclient: get_auth_request con 0x7fcda806db90 auth_method 0 2024-03-29T11:12:49.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.466+0000 7fcdb8ff9640 10 client.4380.objecter ms_handle_connect 0x7fcda806db90 2024-03-29T11:12:49.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.473+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcdb004ef60 con 0x7fcdbc13ff40 2024-03-29T11:12:49.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.473+0000 7fcdb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:49.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.473+0000 7fcdb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:49.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.787+0000 7fcdc243e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.787+0000 7fcdc243e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.787+0000 7fcdc243e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:12:49.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.787+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdbc075f10 con 0x7fcdbc13ff40 2024-03-29T11:12:49.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.788+0000 7fcdb8ff9640 1 -- 172.21.15.105:0/708721121 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fcdb0051060 con 0x7fcdbc13ff40 2024-03-29T11:12:49.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.788+0000 7fcdb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:49.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.788+0000 7fcdb8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:49.767 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:49.767 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.791+0000 7fcdc243e640 10 librados: watch_flush enter 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.791+0000 7fcdc243e640 10 librados: watch_flush exit 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 >> v1:172.21.15.110:6816/1087970004 conn(0x7fcda806db90 legacy=0x7fcda8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 10 monclient: shutdown 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:49.770 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 >> v1:172.21.15.105:6789/0 conn(0x7fcdbc13ff40 legacy=0x7fcdbc1c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:49.771 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 shutdown_connections 2024-03-29T11:12:49.771 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 >> 172.21.15.105:0/708721121 conn(0x7fcdbc0b35e0 msgr2=0x7fcdbc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:49.771 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 shutdown_connections 2024-03-29T11:12:49.771 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 -- 172.21.15.105:0/708721121 wait complete. 2024-03-29T11:12:49.771 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:49.792+0000 7fcdc243e640 1 librados: shutdown 2024-03-29T11:12:49.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:56.597 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:56.755 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.777+0000 7f412971d640 10 monclient: get_monmap_and_config 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.777+0000 7f412971d640 10 monclient: build_initial_monmap 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.777+0000 7f412971d640 10 monclient: monmap: 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:12:56.778368+0000 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:12:56.778368+0000 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:12:56.756 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.757 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.778+0000 7f412971d640 1 Processor -- start 2024-03-29T11:12:56.757 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.778+0000 7f412971d640 1 -- start start 2024-03-29T11:12:56.757 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.778+0000 7f412971d640 10 monclient: init 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient(hunting): picked mon.noname-a con 0x7f41240ae9b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient(hunting): picked mon.noname-c con 0x7f4124143f50 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient(hunting): picked mon.noname-b con 0x7f41240b4890 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240aed90 con 0x7f41240ae9b0 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240af020 con 0x7f4124143f50 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240af300 con 0x7f41240b4890 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f412971d640 10 monclient(hunting): authenticate will time out at 1051.056195s 2024-03-29T11:12:56.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4122ffd640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f41240ae9b0 0x7f4124143840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:47220/0 (socket says 172.21.15.105:47220) 2024-03-29T11:12:56.759 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4122ffd640 1 -- 172.21.15.105:0/1120118898 learned_addr learned my addr 172.21.15.105:0/1120118898 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120187309 0 0) 0x7f41240af020 con 0x7f4124143f50 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c003520 con 0x7f4124143f50 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683370805 0 0) 0x7f41240aed90 con 0x7f41240ae9b0 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.780+0000 7f4121ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41240af020 con 0x7f41240ae9b0 2024-03-29T11:12:56.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598100144 0 0) 0x7f410c003520 con 0x7f4124143f50 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 >> v1:172.21.15.105:6790/0 conn(0x7f41240b4890 legacy=0x7f4124151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 >> v1:172.21.15.105:6789/0 conn(0x7f41240ae9b0 legacy=0x7f4124143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 monclient: found mon.noname-c 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240b0710 con 0x7f4124143f50 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f412971d640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f412971d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:56.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4118001690 con 0x7f4124143f50 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: dump: 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.763 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f412971d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41180018b0 con 0x7f4124143f50 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f4121ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f412971d640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.781+0000 7f412971d640 4 set_mon_vals no callback set 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f4121ffb640 1 -- 172.21.15.105:0/1120118898 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f4118001bc0 con 0x7f4124143f50 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f4121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f4121ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:56.764 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f4121ffb640 10 monclient: dump: 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f412971d640 10 monclient: shutdown 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f412971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f412971d640 1 -- 172.21.15.105:0/1120118898 >> v1:172.21.15.110:6789/0 conn(0x7f4124143f50 legacy=0x7f41240b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f412971d640 1 -- 172.21.15.105:0/1120118898 shutdown_connections 2024-03-29T11:12:56.765 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.782+0000 7f412971d640 1 -- 172.21.15.105:0/1120118898 >> 172.21.15.105:0/1120118898 conn(0x7f4124066a00 msgr2=0x7f4124066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:56.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.783+0000 7f412971d640 1 -- 172.21.15.105:0/1120118898 shutdown_connections 2024-03-29T11:12:56.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.783+0000 7f412971d640 1 -- 172.21.15.105:0/1120118898 wait complete. 2024-03-29T11:12:56.772 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.784+0000 7f412971d640 10 monclient: build_initial_monmap 2024-03-29T11:12:56.772 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.784+0000 7f412971d640 10 monclient: monmap: 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.773 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:12:56.774 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:12:56.774 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:12:56.774 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.774 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.784+0000 7f412971d640 1 librados: starting msgr at 2024-03-29T11:12:56.774 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.784+0000 7f412971d640 1 librados: starting objecter 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 1 Processor -- start 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 1 -- start start 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 1 librados: setting wanted keys 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 1 librados: calling monclient init 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 10 monclient: init 2024-03-29T11:12:56.775 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.785+0000 7f412971d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient(hunting): picked mon.noname-a con 0x7f41240ae9b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient(hunting): picked mon.noname-c con 0x7f41240b4890 addr v1:172.21.15.105:6790/0 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient(hunting): picked mon.noname-b con 0x7f4124143f50 addr v1:172.21.15.110:6789/0 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41241dbb30 con 0x7f41240ae9b0 2024-03-29T11:12:56.776 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240aa890 con 0x7f4124143f50 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41241dbd80 con 0x7f41240b4890 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f412971d640 10 monclient(hunting): authenticate will time out at 1051.062065s 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f41237fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4124143f50 0x7f41241dc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:47792/0 (socket says 172.21.15.105:47792) 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.786+0000 7f41237fe640 1 -- 172.21.15.105:0/141128810 learned_addr learned my addr 172.21.15.105:0/141128810 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter ms_handle_connect 0x7f4124143f50 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter ms_handle_connect 0x7f41240ae9b0 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter ms_handle_connect 0x7f41240b4890 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723155 0 0) 0x7f41240aa890 con 0x7f4124143f50 2024-03-29T11:12:56.777 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110004eb0 con 0x7f4124143f50 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064381036 0 0) 0x7f41241dbb30 con 0x7f41240ae9b0 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.778 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41240aa890 con 0x7f41240ae9b0 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381431107 0 0) 0x7f41241dbd80 con 0x7f41240b4890 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41241dbb30 con 0x7f41240b4890 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749910238 0 0) 0x7f41240aa890 con 0x7f41240ae9b0 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41241dbd80 con 0x7f41240ae9b0 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f41140018b0 con 0x7f41240ae9b0 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:56.779 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.787+0000 7f410b7fe640 10 monclient(hunting): dump: 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:56.780 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310529881 0 0) 0x7f4110004eb0 con 0x7f4124143f50 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41240aa890 con 0x7f4124143f50 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f411c001cd0 con 0x7f4124143f50 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): dump: 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.781 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374705851 0 0) 0x7f41241dbb30 con 0x7f41240b4890 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4110004eb0 con 0x7f41240b4890 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f41180013b0 con 0x7f41240b4890 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 10 monclient(hunting): dump: 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.782 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2378133798 0 0) 0x7f41240aa890 con 0x7f4124143f50 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.788+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 >> v1:172.21.15.105:6790/0 conn(0x7f41240b4890 legacy=0x7f41240aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 >> v1:172.21.15.105:6789/0 conn(0x7f41240ae9b0 legacy=0x7f412413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 1 monclient: found mon.b 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:56.783 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41241dc9b0 con 0x7f4124143f50 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 10 monclient: _renew_subs 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41240aab00 con 0x7f4124143f50 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:56.784 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 10 monclient: _renew_subs 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41240a97e0 con 0x7f4124143f50 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f412971d640 1 librados: init done 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f411c003990 con 0x7f4124143f50 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f4120ff9640 4 set_mon_vals no callback set 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f411c001e30 con 0x7f4124143f50 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.789+0000 7f410b7fe640 10 monclient: dump: 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:12:56.785 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.791+0000 7f412971d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.791+0000 7f412971d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.791+0000 7f412971d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.791+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4104005080 con 0x7f4124143f50 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.792+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f411c003d20 con 0x7f4124143f50 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f411c084d30 con 0x7f4124143f50 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 10 client.4384.objecter ms_dispatch 0x7f4124000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:56.786 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f4122ffd640 10 monclient: get_auth_request con 0x7f411006e280 auth_method 0 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.793+0000 7f410b7fe640 10 client.4384.objecter ms_handle_connect 0x7f411006e280 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.800+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f411c050440 con 0x7f4124143f50 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.800+0000 7f410b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:56.787 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:56.800+0000 7f410b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.182+0000 7f412971d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.182+0000 7f412971d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.182+0000 7f412971d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.182+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4104005330 con 0x7f4124143f50 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.183+0000 7f410b7fe640 1 -- 172.21.15.105:0/141128810 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f411c052540 con 0x7f4124143f50 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.183+0000 7f410b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.183+0000 7f410b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:57.163 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:12:57.164 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.186+0000 7f412971d640 10 librados: watch_flush enter 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 10 librados: watch_flush exit 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 >> v1:172.21.15.110:6816/1087970004 conn(0x7f411006e280 legacy=0x7f4110070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 10 monclient: shutdown 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 >> v1:172.21.15.110:6789/0 conn(0x7f4124143f50 legacy=0x7f41241dc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 shutdown_connections 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 >> 172.21.15.105:0/141128810 conn(0x7f4124066a00 msgr2=0x7f4124066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:57.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.187+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 shutdown_connections 2024-03-29T11:12:57.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.188+0000 7f412971d640 1 -- 172.21.15.105:0/141128810 wait complete. 2024-03-29T11:12:57.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:12:57.188+0000 7f412971d640 1 librados: shutdown 2024-03-29T11:12:57.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:03.990 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:04.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.158+0000 7fc9b11a3640 10 monclient: get_monmap_and_config 2024-03-29T11:13:04.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.158+0000 7fc9b11a3640 10 monclient: build_initial_monmap 2024-03-29T11:13:04.136 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.158+0000 7fc9b11a3640 10 monclient: monmap: 2024-03-29T11:13:04.136 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:04.159521+0000 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:04.159521+0000 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:04.137 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.159+0000 7fc9b11a3640 1 Processor -- start 2024-03-29T11:13:04.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.159+0000 7fc9b11a3640 1 -- start start 2024-03-29T11:13:04.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.159+0000 7fc9b11a3640 10 monclient: init 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a40389e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a4038dc0 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a4030660 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40c54c0 con 0x7fc9a4030660 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4036430 con 0x7fc9a4038dc0 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4039840 con 0x7fc9a40389e0 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.160+0000 7fc9b11a3640 10 monclient(hunting): authenticate will time out at 1058.436904s 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.161+0000 7fc9abfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fc9a40389e0 0x7fc9a4033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:53820/0 (socket says 172.21.15.105:53820) 2024-03-29T11:13:04.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.161+0000 7fc9abfff640 1 -- 172.21.15.105:0/470980335 learned_addr learned my addr 172.21.15.105:0/470980335 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903159800 0 0) 0x7fc9a4039840 con 0x7fc9a40389e0 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:13:04.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc99c003520 con 0x7fc9a40389e0 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589600668 0 0) 0x7fc9a4036430 con 0x7fc9a4038dc0 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a4039840 con 0x7fc9a4038dc0 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506440304 0 0) 0x7fc9a40c54c0 con 0x7fc9a4030660 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:04.141 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.162+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a4036430 con 0x7fc9a4030660 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667257726 0 0) 0x7fc9a4039840 con 0x7fc9a4038dc0 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 >> v1:172.21.15.105:6790/0 conn(0x7fc9a40389e0 legacy=0x7fc9a4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.142 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 >> v1:172.21.15.105:6789/0 conn(0x7fc9a4030660 legacy=0x7fc9a40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 monclient: found mon.noname-c 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4035690 con 0x7fc9a4038dc0 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc998001670 con 0x7fc9a4038dc0 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:04.144 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9aaffd640 10 monclient: dump: 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9b11a3640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:13:04.145 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.163+0000 7fc9b11a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9980017d0 con 0x7fc9a4038dc0 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 1 -- 172.21.15.105:0/470980335 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc998001ba0 con 0x7fc9a4038dc0 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9aaffd640 10 monclient: dump: 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.148 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9b11a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.166+0000 7fc9b11a3640 4 set_mon_vals no callback set 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 10 monclient: shutdown 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:04.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 1 -- 172.21.15.105:0/470980335 >> v1:172.21.15.110:6789/0 conn(0x7fc9a4038dc0 legacy=0x7fc9a40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 1 -- 172.21.15.105:0/470980335 shutdown_connections 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 1 -- 172.21.15.105:0/470980335 >> 172.21.15.105:0/470980335 conn(0x7fc9a4005c00 msgr2=0x7fc9a4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 1 -- 172.21.15.105:0/470980335 shutdown_connections 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.167+0000 7fc9b11a3640 1 -- 172.21.15.105:0/470980335 wait complete. 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.168+0000 7fc9b11a3640 10 monclient: build_initial_monmap 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.168+0000 7fc9b11a3640 10 monclient: monmap: 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:04.150 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 librados: starting msgr at 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 librados: starting objecter 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 Processor -- start 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 -- start start 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 librados: setting wanted keys 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 1 librados: calling monclient init 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.169+0000 7fc9b11a3640 10 monclient: init 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:04.151 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a4030660 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a40389e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a40c2c50 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a402f020 con 0x7fc9a40389e0 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40c29a0 con 0x7fc9a40c2c50 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40c73f0 con 0x7fc9a4030660 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:04.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9b11a3640 10 monclient(hunting): authenticate will time out at 1058.446539s 2024-03-29T11:13:04.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9abfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fc9a4030660 0x7fc9a415fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:54690/0 (socket says 172.21.15.105:54690) 2024-03-29T11:13:04.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9abfff640 1 -- 172.21.15.105:0/2141412883 learned_addr learned my addr 172.21.15.105:0/2141412883 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:04.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc9a4030660 2024-03-29T11:13:04.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.170+0000 7fc9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc9a40c2c50 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075482987 0 0) 0x7fc9a40c73f0 con 0x7fc9a4030660 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc990004eb0 con 0x7fc9a4030660 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc9a40389e0 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:04.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867262470 0 0) 0x7fc9a402f020 con 0x7fc9a40389e0 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.171+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a40c73f0 con 0x7fc9a40389e0 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818245365 0 0) 0x7fc990004eb0 con 0x7fc9a4030660 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a402f020 con 0x7fc9a4030660 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc9a0001950 con 0x7fc9a4030660 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.160 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.172+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787428559 0 0) 0x7fc9a40c73f0 con 0x7fc9a40389e0 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc990004eb0 con 0x7fc9a40389e0 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc998001b20 con 0x7fc9a40389e0 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:04.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:04.162 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180171772 0 0) 0x7fc9a402f020 con 0x7fc9a4030660 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 >> v1:172.21.15.110:6789/0 conn(0x7fc9a40c2c50 legacy=0x7fc9a40c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 >> v1:172.21.15.105:6789/0 conn(0x7fc9a40389e0 legacy=0x7fc9a40c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 monclient: found mon.c 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40c7580 con 0x7fc9a4030660 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.173+0000 7fc9a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:04.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 10 monclient: _renew_subs 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a402e960 con 0x7fc9a4030660 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 10 monclient: _renew_subs 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a402f540 con 0x7fc9a4030660 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9b11a3640 1 librados: init done 2024-03-29T11:13:04.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9a00047e0 con 0x7fc9a4030660 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc9a0004af0 con 0x7fc9a4030660 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.174+0000 7fc9a8ff9640 10 monclient: dump: 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:04.165 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.175+0000 7fc9b11a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.175+0000 7fc9b11a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.175+0000 7fc9b11a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.175+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a4007b90 con 0x7fc9a4030660 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.177+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fc9a001aac0 con 0x7fc9a4030660 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.177+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fc9a0084a30 con 0x7fc9a4030660 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9ab7fe640 10 monclient: get_auth_request con 0x7fc99006de70 auth_method 0 2024-03-29T11:13:04.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9a8ff9640 10 client.4436.objecter ms_dispatch 0x7fc9ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:04.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9a8ff9640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:04.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9a8ff9640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:04.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9a8ff9640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:13:04.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.178+0000 7fc9a8ff9640 10 client.4436.objecter ms_handle_connect 0x7fc99006de70 2024-03-29T11:13:04.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.188+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9a0050310 con 0x7fc9a4030660 2024-03-29T11:13:04.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.188+0000 7fc9a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:04.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.188+0000 7fc9a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:04.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.544+0000 7fc9b11a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.544+0000 7fc9b11a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.544+0000 7fc9b11a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:04.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.544+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a402f1b0 con 0x7fc9a4030660 2024-03-29T11:13:04.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.545+0000 7fc9a8ff9640 1 -- 172.21.15.105:0/2141412883 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fc9a0052240 con 0x7fc9a4030660 2024-03-29T11:13:04.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.545+0000 7fc9a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:04.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.545+0000 7fc9a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:04.523 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:04.523 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 10 librados: watch_flush enter 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 10 librados: watch_flush exit 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 >> v1:172.21.15.110:6816/1087970004 conn(0x7fc99006de70 legacy=0x7fc990070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 10 monclient: shutdown 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.548+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 >> v1:172.21.15.105:6790/0 conn(0x7fc9a4030660 legacy=0x7fc9a415fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.549+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 shutdown_connections 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.549+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 >> 172.21.15.105:0/2141412883 conn(0x7fc9a4005c00 msgr2=0x7fc9a4005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.549+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 shutdown_connections 2024-03-29T11:13:04.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.549+0000 7fc9b11a3640 1 -- 172.21.15.105:0/2141412883 wait complete. 2024-03-29T11:13:04.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:04.549+0000 7fc9b11a3640 1 librados: shutdown 2024-03-29T11:13:04.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:11.352 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.511+0000 7f6534d13640 10 monclient: get_monmap_and_config 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.511+0000 7f6534d13640 10 monclient: build_initial_monmap 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.511+0000 7f6534d13640 10 monclient: monmap: 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:11.512227+0000 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:11.512227+0000 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:11.489 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:11.490 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:11.490 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.490 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.512+0000 7f6534d13640 1 Processor -- start 2024-03-29T11:13:11.490 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.512+0000 7f6534d13640 1 -- start start 2024-03-29T11:13:11.490 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.512+0000 7f6534d13640 10 monclient: init 2024-03-29T11:13:11.491 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.512+0000 7f6534d13640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-a con 0x7f65301457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-c con 0x7f653013f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-b con 0x7f65300a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300aaef0 con 0x7f65301457f0 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530145bd0 con 0x7f653013f650 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530145eb0 con 0x7f65300a9d70 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f6534d13640 10 monclient(hunting): authenticate will time out at 1065.789651s 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f652ed9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f65300a9d70 0x7f653014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:57200/0 (socket says 172.21.15.105:57200) 2024-03-29T11:13:11.492 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.513+0000 7f652ed9d640 1 -- 172.21.15.105:0/2791445531 learned_addr learned my addr 172.21.15.105:0/2791445531 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:11.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355278966 0 0) 0x7f6530145bd0 con 0x7f653013f650 2024-03-29T11:13:11.496 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6510003520 con 0x7f653013f650 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938406836 0 0) 0x7f6530145eb0 con 0x7f65300a9d70 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.514+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6530145bd0 con 0x7f65300a9d70 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.515+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244974473 0 0) 0x7f6530145bd0 con 0x7f65300a9d70 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.515+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 >> v1:172.21.15.110:6789/0 conn(0x7f653013f650 legacy=0x7f65300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.515+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 >> v1:172.21.15.105:6789/0 conn(0x7f65301457f0 legacy=0x7f653013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:11.497 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6530146340 con 0x7f65300a9d70 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.516+0000 7f652dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f65280018a0 con 0x7f65300a9d70 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: dump: 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.498 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6528001ac0 con 0x7f65300a9d70 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 1 -- 172.21.15.105:0/2791445531 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6528001e50 con 0x7f65300a9d70 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f652dd9b640 10 monclient: dump: 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:11.499 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f6534d13640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f6534d13640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.517+0000 7f6534d13640 4 set_mon_vals no callback set 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 10 monclient: shutdown 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 1 -- 172.21.15.105:0/2791445531 >> v1:172.21.15.105:6790/0 conn(0x7f65300a9d70 legacy=0x7f653014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 1 -- 172.21.15.105:0/2791445531 shutdown_connections 2024-03-29T11:13:11.501 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 1 -- 172.21.15.105:0/2791445531 >> 172.21.15.105:0/2791445531 conn(0x7f6530066a00 msgr2=0x7f6530066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:11.501 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 1 -- 172.21.15.105:0/2791445531 shutdown_connections 2024-03-29T11:13:11.501 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.518+0000 7f6534d13640 1 -- 172.21.15.105:0/2791445531 wait complete. 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.519+0000 7f6534d13640 10 monclient: build_initial_monmap 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.519+0000 7f6534d13640 10 monclient: monmap: 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:11.519 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.519+0000 7f6534d13640 1 librados: starting msgr at 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.519+0000 7f6534d13640 1 librados: starting objecter 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.520+0000 7f6534d13640 1 Processor -- start 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.520+0000 7f6534d13640 1 -- start start 2024-03-29T11:13:11.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.520+0000 7f6534d13640 1 librados: setting wanted keys 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.520+0000 7f6534d13640 1 librados: calling monclient init 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.520+0000 7f6534d13640 10 monclient: init 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6534d13640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6534d13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-c con 0x7f65300a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-b con 0x7f653013f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f652e59c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f65300a9d70 0x7f653013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:58268/0 (socket says 172.21.15.105:58268) 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f652e59c640 1 -- 172.21.15.105:0/2807853510 learned_addr learned my addr 172.21.15.105:0/2807853510 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6526ffd640 10 client.?.objecter ms_handle_connect 0x7f65300a9d70 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6526ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6534d13640 10 monclient(hunting): picked mon.noname-a con 0x7f65301457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6526ffd640 10 client.?.objecter ms_handle_connect 0x7f653013f650 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.521+0000 7f6526ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:11.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653013b9f0 con 0x7f65301457f0 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530142e70 con 0x7f653013f650 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530141ef0 con 0x7f65300a9d70 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6534d13640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275650055 0 0) 0x7f6530142e70 con 0x7f653013f650 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6534d13640 10 monclient(hunting): authenticate will time out at 1065.798452s 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6514004eb0 con 0x7f653013f650 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614808102 0 0) 0x7f6530141ef0 con 0x7f65300a9d70 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.522+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6530142e70 con 0x7f65300a9d70 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952162921 0 0) 0x7f6514004eb0 con 0x7f653013f650 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6530141ef0 con 0x7f653013f650 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6518001970 con 0x7f653013f650 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:11.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056156830 0 0) 0x7f6530142e70 con 0x7f65300a9d70 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6514004eb0 con 0x7f65300a9d70 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6520001710 con 0x7f65300a9d70 2024-03-29T11:13:11.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 179284359 0 0) 0x7f6530141ef0 con 0x7f653013f650 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.523+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 >> v1:172.21.15.105:6790/0 conn(0x7f65300a9d70 legacy=0x7f653013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 >> v1:172.21.15.105:6789/0 conn(0x7f65301457f0 legacy=0x7f65301e5de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:11.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 1 monclient: found mon.b 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65301358e0 con 0x7f653013f650 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6526ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 10 monclient: _renew_subs 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6530140c20 con 0x7f653013f650 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 10 monclient: _renew_subs 2024-03-29T11:13:11.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f653013b740 con 0x7f653013f650 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.524+0000 7f6534d13640 1 librados: init done 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.525+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6518001ad0 con 0x7f653013f650 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.525+0000 7f6526ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.525+0000 7f652d59a640 4 set_mon_vals no callback set 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.526+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6518003a60 con 0x7f653013f650 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.526+0000 7f6534d13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.526+0000 7f6534d13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.526+0000 7f6534d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.526+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6530075f10 con 0x7f653013f650 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.527+0000 7f6526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.527+0000 7f6526ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.527+0000 7f6526ffd640 10 monclient: dump: 2024-03-29T11:13:11.527 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.527+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f6518019a30 con 0x7f653013f650 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f6518083880 con 0x7f653013f650 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 10 client.4396.objecter ms_dispatch 0x7f6530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:11.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T11:13:11.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f652e59c640 10 monclient: get_auth_request con 0x7f651406de40 auth_method 0 2024-03-29T11:13:11.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.528+0000 7f6526ffd640 10 client.4396.objecter ms_handle_connect 0x7f651406de40 2024-03-29T11:13:11.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.535+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f651804f180 con 0x7f653013f650 2024-03-29T11:13:11.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.535+0000 7f6526ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:11.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.535+0000 7f6526ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:11.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.840+0000 7f6534d13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.840+0000 7f6534d13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.840+0000 7f6534d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:13:11.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.840+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65300b3680 con 0x7f653013f650 2024-03-29T11:13:11.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.844+0000 7f6526ffd640 1 -- 172.21.15.105:0/2807853510 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f65180510b0 con 0x7f653013f650 2024-03-29T11:13:11.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.844+0000 7f6526ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:11.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.844+0000 7f6526ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:11.822 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:11.822 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:11.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 10 librados: watch_flush enter 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 10 librados: watch_flush exit 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 10 client.4396.objecter successfully canceled tick 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 >> v1:172.21.15.110:6816/1087970004 conn(0x7f651406de40 legacy=0x7f6514070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 10 monclient: shutdown 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 >> v1:172.21.15.110:6789/0 conn(0x7f653013f650 legacy=0x7f6530142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 shutdown_connections 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 >> 172.21.15.105:0/2807853510 conn(0x7f6530066a00 msgr2=0x7f65300b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 shutdown_connections 2024-03-29T11:13:11.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 -- 172.21.15.105:0/2807853510 wait complete. 2024-03-29T11:13:11.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:11.847+0000 7f6534d13640 1 librados: shutdown 2024-03-29T11:13:11.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:18.650 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:18.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.821+0000 7ff71165f640 10 monclient: get_monmap_and_config 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.821+0000 7ff71165f640 10 monclient: build_initial_monmap 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.821+0000 7ff71165f640 10 monclient: monmap: 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:18.822809+0000 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:18.822809+0000 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:18.800 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.823+0000 7ff71165f640 1 Processor -- start 2024-03-29T11:13:18.803 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.823+0000 7ff71165f640 1 -- start start 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.823+0000 7ff71165f640 10 monclient: init 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-c con 0x7ff7040389e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-a con 0x7ff704038dc0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-b con 0x7ff704030660 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040c54c0 con 0x7ff704038dc0 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff704036430 con 0x7ff7040389e0 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff704039840 con 0x7ff704030660 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:18.804 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.824+0000 7ff71165f640 10 monclient(hunting): authenticate will time out at 1073.100540s 2024-03-29T11:13:18.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.830+0000 7ff70b7fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff704038dc0 0x7ff7040cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:54113/0 (socket says 172.21.15.105:54113) 2024-03-29T11:13:18.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.830+0000 7ff70b7fe640 1 -- 172.21.15.105:0/1215538429 learned_addr learned my addr 172.21.15.105:0/1215538429 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:18.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.832+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757324736 0 0) 0x7ff7040c54c0 con 0x7ff704038dc0 2024-03-29T11:13:18.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.832+0000 7ff70affd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc003520 con 0x7ff704038dc0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768723962 0 0) 0x7ff704036430 con 0x7ff7040389e0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7040c54c0 con 0x7ff7040389e0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381622966 0 0) 0x7ff6fc003520 con 0x7ff704038dc0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 >> v1:172.21.15.105:6790/0 conn(0x7ff704030660 legacy=0x7ff7040d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 >> v1:172.21.15.110:6789/0 conn(0x7ff7040389e0 legacy=0x7ff704033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 monclient: found mon.noname-a 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:13:18.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704035690 con 0x7ff704038dc0 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: _finish_auth 0 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff71165f640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff71165f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff700001690 con 0x7ff704038dc0 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:18.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff70affd640 10 monclient: dump: 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.833+0000 7ff71165f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.834+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff700001860 con 0x7ff704038dc0 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.834+0000 7ff70affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.834+0000 7ff71165f640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:18.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.834+0000 7ff71165f640 4 set_mon_vals no callback set 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.834+0000 7ff70affd640 1 -- 172.21.15.105:0/1215538429 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff700001c70 con 0x7ff704038dc0 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff70affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 10 monclient: shutdown 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff70affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff70affd640 10 monclient: dump: 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:18.818 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 1 -- 172.21.15.105:0/1215538429 >> v1:172.21.15.105:6789/0 conn(0x7ff704038dc0 legacy=0x7ff7040cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 1 -- 172.21.15.105:0/1215538429 shutdown_connections 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 1 -- 172.21.15.105:0/1215538429 >> 172.21.15.105:0/1215538429 conn(0x7ff704005c00 msgr2=0x7ff704006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:18.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.835+0000 7ff71165f640 1 -- 172.21.15.105:0/1215538429 shutdown_connections 2024-03-29T11:13:18.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.836+0000 7ff71165f640 1 -- 172.21.15.105:0/1215538429 wait complete. 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.836+0000 7ff71165f640 10 monclient: build_initial_monmap 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.837+0000 7ff71165f640 10 monclient: monmap: 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.837+0000 7ff71165f640 1 librados: starting msgr at 2024-03-29T11:13:18.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.837+0000 7ff71165f640 1 librados: starting objecter 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.837+0000 7ff71165f640 1 Processor -- start 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 1 -- start start 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 1 librados: setting wanted keys 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 1 librados: calling monclient init 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 10 monclient: init 2024-03-29T11:13:18.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:18.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.838+0000 7ff71165f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:18.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.839+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-c con 0x7ff704030660 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:18.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.839+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-a con 0x7ff7040389e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:18.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 10 monclient(hunting): picked mon.noname-b con 0x7ff7040c5650 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70402e260 con 0x7ff7040389e0 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70402e4d0 con 0x7ff7040c5650 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040c7850 con 0x7ff704030660 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.843+0000 7ff71165f640 10 monclient(hunting): authenticate will time out at 1073.119405s 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.844+0000 7ff710e5e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff7040c5650 0x7ff704173ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:51771/0 (socket says 172.21.15.105:51771) 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.844+0000 7ff710e5e640 1 -- 172.21.15.105:0/1613528137 learned_addr learned my addr 172.21.15.105:0/1613528137 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.844+0000 7ff708ff9640 10 client.?.objecter ms_handle_connect 0x7ff704030660 2024-03-29T11:13:18.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.844+0000 7ff708ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:18.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871376412 0 0) 0x7ff7040c7850 con 0x7ff704030660 2024-03-29T11:13:18.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:18.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0004eb0 con 0x7ff704030660 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716399414 0 0) 0x7ff6f0004eb0 con 0x7ff704030660 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7040c7850 con 0x7ff704030660 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff70c04fdd0 con 0x7ff704030660 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.833 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4091983186 0 0) 0x7ff7040c7850 con 0x7ff704030660 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 >> v1:172.21.15.110:6789/0 conn(0x7ff7040c5650 legacy=0x7ff704173ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.845+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 >> v1:172.21.15.105:6789/0 conn(0x7ff7040389e0 legacy=0x7ff70416d310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 1 monclient: found mon.c 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:18.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70402d6d0 con 0x7ff704030660 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff708ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff71165f640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff71165f640 10 monclient: _renew_subs 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff71165f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.846+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7040c8360 con 0x7ff704030660 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.847+0000 7ff71165f640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.847+0000 7ff71165f640 10 monclient: _renew_subs 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.847+0000 7ff71165f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.847+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7041755f0 con 0x7ff704030660 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.847+0000 7ff71165f640 1 librados: init done 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.848+0000 7ff71165f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.848+0000 7ff71165f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.848+0000 7ff71165f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.848+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff704007b90 con 0x7ff704030660 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff70c04f400 con 0x7ff704030660 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff70c075710 con 0x7ff704030660 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff708ff9640 10 monclient: dump: 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:18.836 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:18.837 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.850+0000 7ff70a7fc640 4 set_mon_vals no callback set 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff70c067bf0 con 0x7ff704030660 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff70c0fe3a0 con 0x7ff704030660 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 10 client.4448.objecter ms_dispatch 0x7ff70c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.851+0000 7ff708ff9640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.853+0000 7ff70b7fe640 10 monclient: get_auth_request con 0x7ff6f006db10 auth_method 0 2024-03-29T11:13:18.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.854+0000 7ff708ff9640 10 client.4448.objecter ms_handle_connect 0x7ff6f006db10 2024-03-29T11:13:18.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.861+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff70c0c9c80 con 0x7ff704030660 2024-03-29T11:13:18.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.861+0000 7ff708ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:18.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:18.861+0000 7ff708ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:19.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.182+0000 7ff71165f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.182+0000 7ff71165f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.182+0000 7ff71165f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:19.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.182+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff704005120 con 0x7ff704030660 2024-03-29T11:13:19.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.183+0000 7ff708ff9640 1 -- 172.21.15.105:0/1613528137 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff70c0cbbb0 con 0x7ff704030660 2024-03-29T11:13:19.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.183+0000 7ff708ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:19.161 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.183+0000 7ff708ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:19.161 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:19.161 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:19.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.188+0000 7ff71165f640 10 librados: watch_flush enter 2024-03-29T11:13:19.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.189+0000 7ff71165f640 10 librados: watch_flush exit 2024-03-29T11:13:19.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.189+0000 7ff71165f640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:13:19.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.189+0000 7ff71165f640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:13:19.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff6f006db10 legacy=0x7ff6f006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 10 monclient: shutdown 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 >> v1:172.21.15.105:6790/0 conn(0x7ff704030660 legacy=0x7ff70416cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 shutdown_connections 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 >> 172.21.15.105:0/1613528137 conn(0x7ff704005c00 msgr2=0x7ff7040c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.190+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 shutdown_connections 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.191+0000 7ff71165f640 1 -- 172.21.15.105:0/1613528137 wait complete. 2024-03-29T11:13:19.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:19.191+0000 7ff71165f640 1 librados: shutdown 2024-03-29T11:13:19.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:25.995 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.167+0000 7f74b9425640 10 monclient: get_monmap_and_config 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.167+0000 7f74b9425640 10 monclient: build_initial_monmap 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.167+0000 7f74b9425640 10 monclient: monmap: 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.146 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:26.168399+0000 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:26.168399+0000 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.168+0000 7f74b9425640 1 Processor -- start 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.168+0000 7f74b9425640 1 -- start start 2024-03-29T11:13:26.147 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.168+0000 7f74b9425640 10 monclient: init 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.169+0000 7f74b9425640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.169+0000 7f74b9425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.169+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-c con 0x7f74b41457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-a con 0x7f74b413f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-b con 0x7f74b40a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40aaef0 con 0x7f74b413f650 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b4145bd0 con 0x7f74b41457f0 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b4145eb0 con 0x7f74b40a9d70 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:26.149 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.171+0000 7f74b9425640 10 monclient(hunting): authenticate will time out at 1080.447376s 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f74b41457f0 0x7f74b413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:42929/0 (socket says 172.21.15.105:42929) 2024-03-29T11:13:26.152 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b2ffd640 1 -- 172.21.15.105:0/3330926961 learned_addr learned my addr 172.21.15.105:0/3330926961 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690428778 0 0) 0x7f74b4145bd0 con 0x7f74b41457f0 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:13:26.153 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.174+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c003520 con 0x7f74b41457f0 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695888709 0 0) 0x7f74b40aaef0 con 0x7f74b413f650 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c003840 con 0x7f74b413f650 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1320046103 0 0) 0x7f749c003520 con 0x7f74b41457f0 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 >> v1:172.21.15.105:6790/0 conn(0x7f74b40a9d70 legacy=0x7f74b414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 >> v1:172.21.15.105:6789/0 conn(0x7f74b413f650 legacy=0x7f74b40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:13:26.155 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4146340 con 0x7f74b41457f0 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f74ac001970 con 0x7f74b41457f0 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b1ffb640 10 monclient: dump: 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.156 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b9425640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:13:26.157 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.176+0000 7f74b9425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:26.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.177+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74ac001ad0 con 0x7f74b41457f0 2024-03-29T11:13:26.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.177+0000 7f74b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:26.159 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.177+0000 7f74b9425640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.177+0000 7f74b9425640 4 set_mon_vals no callback set 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b1ffb640 1 -- 172.21.15.105:0/3330926961 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f74ac001f20 con 0x7f74b41457f0 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b1ffb640 10 monclient: dump: 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.160 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.161 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.162 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b9425640 10 monclient: shutdown 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b9425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b9425640 1 -- 172.21.15.105:0/3330926961 >> v1:172.21.15.110:6789/0 conn(0x7f74b41457f0 legacy=0x7f74b413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b9425640 1 -- 172.21.15.105:0/3330926961 shutdown_connections 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.178+0000 7f74b9425640 1 -- 172.21.15.105:0/3330926961 >> 172.21.15.105:0/3330926961 conn(0x7f74b4066a00 msgr2=0x7f74b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.179+0000 7f74b9425640 1 -- 172.21.15.105:0/3330926961 shutdown_connections 2024-03-29T11:13:26.163 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.179+0000 7f74b9425640 1 -- 172.21.15.105:0/3330926961 wait complete. 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.180+0000 7f74b9425640 10 monclient: build_initial_monmap 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.180+0000 7f74b9425640 10 monclient: monmap: 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:26.164 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.180+0000 7f74b9425640 1 librados: starting msgr at 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.180+0000 7f74b9425640 1 librados: starting objecter 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.181+0000 7f74b9425640 1 Processor -- start 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.181+0000 7f74b9425640 1 -- start start 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.181+0000 7f74b9425640 1 librados: setting wanted keys 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.181+0000 7f74b9425640 1 librados: calling monclient init 2024-03-29T11:13:26.165 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.181+0000 7f74b9425640 10 monclient: init 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-c con 0x7f74b40a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-b con 0x7f74b413f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient(hunting): picked mon.noname-a con 0x7f74b4142bb0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b41424a0 con 0x7f74b4142bb0 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b41cc7a0 con 0x7f74b413f650 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b41cca80 con 0x7f74b40a9d70 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.182+0000 7f74b9425640 10 monclient(hunting): authenticate will time out at 1080.458803s 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.186+0000 7f74b2ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f74b40a9d70 0x7f74b4140080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:51141/0 (socket says 172.21.15.105:51141) 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.186+0000 7f74b2ffd640 1 -- 172.21.15.105:0/1041191826 learned_addr learned my addr 172.21.15.105:0/1041191826 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:26.166 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.186+0000 7f749b7fe640 10 client.?.objecter ms_handle_connect 0x7f74b40a9d70 2024-03-29T11:13:26.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.187+0000 7f749b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 10 client.?.objecter ms_handle_connect 0x7f74b413f650 2024-03-29T11:13:26.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 10 client.?.objecter ms_handle_connect 0x7f74b4142bb0 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592298571 0 0) 0x7f74b41cc7a0 con 0x7f74b413f650 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.189+0000 7f749b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a0004eb0 con 0x7f74b413f650 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723168406 0 0) 0x7f74b41cca80 con 0x7f74b40a9d70 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:13:26.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b41cc7a0 con 0x7f74b40a9d70 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927517525 0 0) 0x7f74b41424a0 con 0x7f74b4142bb0 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b41cca80 con 0x7f74b4142bb0 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 681751185 0 0) 0x7f74a0004eb0 con 0x7f74b413f650 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b41424a0 con 0x7f74b413f650 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f74a80017f0 con 0x7f74b413f650 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:26.178 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:26.179 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.190+0000 7f749b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3372903684 0 0) 0x7f74b41cc7a0 con 0x7f74b40a9d70 2024-03-29T11:13:26.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74a0004eb0 con 0x7f74b40a9d70 2024-03-29T11:13:26.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f74ac001de0 con 0x7f74b40a9d70 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.185 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.186 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.186 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2203835003 0 0) 0x7f74a0004eb0 con 0x7f74b40a9d70 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 >> v1:172.21.15.110:6789/0 conn(0x7f74b413f650 legacy=0x7f74b4140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 >> v1:172.21.15.105:6789/0 conn(0x7f74b4142bb0 legacy=0x7f74b41e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 monclient: found mon.c 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b41420a0 con 0x7f74b40a9d70 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:26.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f749b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 10 monclient: _renew_subs 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b41cb5b0 con 0x7f74b40a9d70 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 10 monclient: _renew_subs 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b406de90 con 0x7f74b40a9d70 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.191+0000 7f74b9425640 1 librados: init done 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.193+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74ac001f40 con 0x7f74b40a9d70 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.193+0000 7f749b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.193+0000 7f74b17fa640 4 set_mon_vals no callback set 2024-03-29T11:13:26.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f74ac004fb0 con 0x7f74b40a9d70 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 10 monclient: dump: 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.189 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:26.190 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:26.190 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f74ac01af80 con 0x7f74b40a9d70 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f74b27fc640 10 monclient: get_auth_request con 0x7f74a0076870 auth_method 0 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f74ac084df0 con 0x7f74b40a9d70 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 10 client.4451.objecter ms_dispatch 0x7f74b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.194+0000 7f749b7fe640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f749b7fe640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f749b7fe640 10 client.4451.objecter ms_handle_connect 0x7f74a0076870 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f74b9425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f74b9425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f74b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:26.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.195+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7494005080 con 0x7f74b40a9d70 2024-03-29T11:13:26.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.206+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74ac050810 con 0x7f74b40a9d70 2024-03-29T11:13:26.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.206+0000 7f749b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:26.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.206+0000 7f749b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:26.510 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.533+0000 7f74b9425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.533+0000 7f74b9425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.533+0000 7f74b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:26.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.533+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7494005330 con 0x7f74b40a9d70 2024-03-29T11:13:26.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.536+0000 7f749b7fe640 1 -- 172.21.15.105:0/1041191826 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f74ac052740 con 0x7f74b40a9d70 2024-03-29T11:13:26.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.536+0000 7f749b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:26.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.536+0000 7f749b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:26.515 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:26.515 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:26.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 10 librados: watch_flush enter 2024-03-29T11:13:26.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 10 librados: watch_flush exit 2024-03-29T11:13:26.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:13:26.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:13:26.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 >> v1:172.21.15.110:6816/1087970004 conn(0x7f74a0076870 legacy=0x7f74a0078d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 10 monclient: shutdown 2024-03-29T11:13:26.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:26.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.541+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 >> v1:172.21.15.105:6790/0 conn(0x7f74b40a9d70 legacy=0x7f74b4140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:26.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.544+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 shutdown_connections 2024-03-29T11:13:26.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.544+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 >> 172.21.15.105:0/1041191826 conn(0x7f74b4066a00 msgr2=0x7f74b413ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:26.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.544+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 shutdown_connections 2024-03-29T11:13:26.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.545+0000 7f74b9425640 1 -- 172.21.15.105:0/1041191826 wait complete. 2024-03-29T11:13:26.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:26.545+0000 7f74b9425640 1 librados: shutdown 2024-03-29T11:13:26.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:33.346 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.520+0000 7f327d2d6640 10 monclient: get_monmap_and_config 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.520+0000 7f327d2d6640 10 monclient: build_initial_monmap 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.520+0000 7f327d2d6640 10 monclient: monmap: 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:33.521873+0000 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:33.521873+0000 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:33.500 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.501 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:33.501 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:33.501 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:33.501 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.501 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.522+0000 7f327d2d6640 1 Processor -- start 2024-03-29T11:13:33.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.522+0000 7f327d2d6640 1 -- start start 2024-03-29T11:13:33.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.522+0000 7f327d2d6640 10 monclient: init 2024-03-29T11:13:33.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:33.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:33.502 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f32780acf60 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f327813ff40 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f32781403f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b4310 con 0x7f327813ff40 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3278140800 con 0x7f32780acf60 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780aec00 con 0x7f32781403f0 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:33.503 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.523+0000 7f327d2d6640 10 monclient(hunting): authenticate will time out at 1087.799620s 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.526+0000 7f32777fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f32781403f0 0x7f327814bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41245/0 (socket says 172.21.15.105:41245) 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.526+0000 7f32777fe640 1 -- 172.21.15.105:0/3018727265 learned_addr learned my addr 172.21.15.105:0/3018727265 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391093698 0 0) 0x7f32780aec00 con 0x7f32781403f0 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:13:33.507 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.527+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3264003520 con 0x7f32781403f0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626966195 0 0) 0x7f3264003520 con 0x7f32781403f0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 >> v1:172.21.15.110:6789/0 conn(0x7f32780acf60 legacy=0x7f3278139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 >> v1:172.21.15.105:6789/0 conn(0x7f327813ff40 legacy=0x7f3278148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 monclient: found mon.noname-b 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780aeee0 con 0x7f32781403f0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:33.509 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f32700019f0 con 0x7f32781403f0 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f32767fc640 10 monclient: dump: 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:33.510 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:33.511 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f327d2d6640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:13:33.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.530+0000 7f327d2d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3270001b50 con 0x7f32781403f0 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f327d2d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 1 -- 172.21.15.105:0/3018727265 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3270001fa0 con 0x7f32781403f0 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f327d2d6640 4 set_mon_vals no callback set 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.531+0000 7f32767fc640 10 monclient: dump: 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.532+0000 7f327d2d6640 10 monclient: shutdown 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.532+0000 7f327d2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:33.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.532+0000 7f327d2d6640 1 -- 172.21.15.105:0/3018727265 >> v1:172.21.15.105:6790/0 conn(0x7f32781403f0 legacy=0x7f327814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.534+0000 7f327d2d6640 1 -- 172.21.15.105:0/3018727265 shutdown_connections 2024-03-29T11:13:33.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.534+0000 7f327d2d6640 1 -- 172.21.15.105:0/3018727265 >> 172.21.15.105:0/3018727265 conn(0x7f32780b35e0 msgr2=0x7f32780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.535+0000 7f327d2d6640 1 -- 172.21.15.105:0/3018727265 shutdown_connections 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.535+0000 7f327d2d6640 1 -- 172.21.15.105:0/3018727265 wait complete. 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 10 monclient: build_initial_monmap 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 10 monclient: monmap: 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:33.515 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 librados: starting msgr at 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 librados: starting objecter 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 Processor -- start 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 -- start start 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 librados: setting wanted keys 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 1 librados: calling monclient init 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.536+0000 7f327d2d6640 10 monclient: init 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:33.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f32780acf60 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f327813ff40 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f32781403f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327813c900 con 0x7f32780acf60 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327813cbe0 con 0x7f32781403f0 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327813a7d0 con 0x7f327813ff40 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:33.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.537+0000 7f327d2d6640 10 monclient(hunting): authenticate will time out at 1087.813482s 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326ffff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f327813ff40 0x7f32781ec070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41347/0 (socket says 172.21.15.105:41347) 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326ffff640 1 -- 172.21.15.105:0/2910219631 learned_addr learned my addr 172.21.15.105:0/2910219631 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326f7fe640 10 client.?.objecter ms_handle_connect 0x7f32780acf60 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326f7fe640 10 client.?.objecter ms_handle_connect 0x7f327813ff40 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.543+0000 7f326f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:33.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666488335 0 0) 0x7f327813c900 con 0x7f32780acf60 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3258004eb0 con 0x7f32780acf60 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713856520 0 0) 0x7f327813a7d0 con 0x7f327813ff40 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327813c900 con 0x7f327813ff40 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808182336 0 0) 0x7f3258004eb0 con 0x7f32780acf60 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f327813a7d0 con 0x7f32780acf60 2024-03-29T11:13:33.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f32680018f0 con 0x7f32780acf60 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 client.?.objecter ms_handle_connect 0x7f32781403f0 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.544+0000 7f326f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 830030121 0 0) 0x7f327813c900 con 0x7f327813ff40 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3258004eb0 con 0x7f327813ff40 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f32600019f0 con 0x7f327813ff40 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.530 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3448873379 0 0) 0x7f327813a7d0 con 0x7f32780acf60 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 >> v1:172.21.15.105:6790/0 conn(0x7f327813ff40 legacy=0x7f32781ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 >> v1:172.21.15.110:6789/0 conn(0x7f32781403f0 legacy=0x7f32781ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 monclient: found mon.a 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327806e7a0 con 0x7f32780acf60 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:33.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f326f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 10 monclient: _renew_subs 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32781278a0 con 0x7f32780acf60 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 10 monclient: _renew_subs 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32781374a0 con 0x7f32780acf60 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.545+0000 7f327d2d6640 1 librados: init done 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32680035f0 con 0x7f32780acf60 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3268003900 con 0x7f32780acf60 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:33.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 10 monclient: dump: 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.547+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f32680198d0 con 0x7f32780acf60 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f3268083920 con 0x7f32780acf60 2024-03-29T11:13:33.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f326f7fe640 10 client.4407.objecter ms_dispatch 0x7f3278000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:33.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f326f7fe640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:33.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f326f7fe640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:33.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f326f7fe640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:13:33.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.548+0000 7f32757fa640 4 set_mon_vals no callback set 2024-03-29T11:13:33.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.549+0000 7f327d2d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.550+0000 7f327d2d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.550+0000 7f327d2d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:33.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.550+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c005080 con 0x7f32780acf60 2024-03-29T11:13:33.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.550+0000 7f326ffff640 10 monclient: get_auth_request con 0x7f325806df30 auth_method 0 2024-03-29T11:13:33.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.551+0000 7f326f7fe640 10 client.4407.objecter ms_handle_connect 0x7f325806df30 2024-03-29T11:13:33.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.559+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f326804f280 con 0x7f32780acf60 2024-03-29T11:13:33.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.559+0000 7f326f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:33.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.559+0000 7f326f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:33.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.879+0000 7f327d2d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.879+0000 7f327d2d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.879+0000 7f327d2d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:33.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.879+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f325c005330 con 0x7f32780acf60 2024-03-29T11:13:33.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.880+0000 7f326f7fe640 1 -- 172.21.15.105:0/2910219631 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f32680511b0 con 0x7f32780acf60 2024-03-29T11:13:33.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.880+0000 7f326f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:33.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.880+0000 7f326f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:33.861 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:33.861 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 10 librados: watch_flush enter 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 10 librados: watch_flush exit 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 >> v1:172.21.15.110:6816/1087970004 conn(0x7f325806df30 legacy=0x7f32580703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 10 monclient: shutdown 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 >> v1:172.21.15.105:6789/0 conn(0x7f32780acf60 legacy=0x7f327813d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:33.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 shutdown_connections 2024-03-29T11:13:33.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 >> 172.21.15.105:0/2910219631 conn(0x7f32780b35e0 msgr2=0x7f32780b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:33.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 shutdown_connections 2024-03-29T11:13:33.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.883+0000 7f327d2d6640 1 -- 172.21.15.105:0/2910219631 wait complete. 2024-03-29T11:13:33.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:33.884+0000 7f327d2d6640 1 librados: shutdown 2024-03-29T11:13:33.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:40.687 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:40.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.864+0000 7f889a416640 10 monclient: get_monmap_and_config 2024-03-29T11:13:40.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.864+0000 7f889a416640 10 monclient: build_initial_monmap 2024-03-29T11:13:40.842 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.864+0000 7f889a416640 10 monclient: monmap: 2024-03-29T11:13:40.842 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:40.865692+0000 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:40.865692+0000 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:40.843 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.865+0000 7f889a416640 1 Processor -- start 2024-03-29T11:13:40.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.865+0000 7f889a416640 1 -- start start 2024-03-29T11:13:40.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.865+0000 7f889a416640 10 monclient: init 2024-03-29T11:13:40.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.866+0000 7f889a416640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:40.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.866+0000 7f889a416640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:40.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.866+0000 7f889a416640 10 monclient(hunting): picked mon.noname-b con 0x7f88941457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:40.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.866+0000 7f889a416640 10 monclient(hunting): picked mon.noname-c con 0x7f889413f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:40.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.871+0000 7f889a416640 10 monclient(hunting): picked mon.noname-a con 0x7f88940a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:40.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.871+0000 7f889a416640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940aaef0 con 0x7f88940a9d70 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.871+0000 7f889a416640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894145bd0 con 0x7f889413f650 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.872+0000 7f889a416640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894145eb0 con 0x7f88941457f0 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.872+0000 7f889a416640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.872+0000 7f889a416640 10 monclient(hunting): authenticate will time out at 1095.147991s 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.872+0000 7f8893fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f88941457f0 0x7f889413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:40007/0 (socket says 172.21.15.105:40007) 2024-03-29T11:13:40.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.872+0000 7f8893fff640 1 -- 172.21.15.105:0/3570492162 learned_addr learned my addr 172.21.15.105:0/3570492162 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894111013 0 0) 0x7f8894145eb0 con 0x7f88941457f0 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f887c003520 con 0x7f88941457f0 2024-03-29T11:13:40.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2074756827 0 0) 0x7f887c003520 con 0x7f88941457f0 2024-03-29T11:13:40.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 >> v1:172.21.15.110:6789/0 conn(0x7f889413f650 legacy=0x7f88940b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:13:40.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 >> v1:172.21.15.105:6789/0 conn(0x7f88940a9d70 legacy=0x7f889414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:40.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 monclient: found mon.noname-b 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894146340 con 0x7f88941457f0 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.873+0000 7f8892ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8884001690 con 0x7f88941457f0 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:40.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: dump: 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f889a416640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f889a416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8884001860 con 0x7f88941457f0 2024-03-29T11:13:40.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 1 -- 172.21.15.105:0/3570492162 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8884001cb0 con 0x7f88941457f0 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f8892ffd640 10 monclient: dump: 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:40.855 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f889a416640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.874+0000 7f889a416640 4 set_mon_vals no callback set 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 10 monclient: shutdown 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 1 -- 172.21.15.105:0/3570492162 >> v1:172.21.15.105:6790/0 conn(0x7f88941457f0 legacy=0x7f889413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 1 -- 172.21.15.105:0/3570492162 shutdown_connections 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 1 -- 172.21.15.105:0/3570492162 >> 172.21.15.105:0/3570492162 conn(0x7f8894066a00 msgr2=0x7f8894066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 1 -- 172.21.15.105:0/3570492162 shutdown_connections 2024-03-29T11:13:40.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.875+0000 7f889a416640 1 -- 172.21.15.105:0/3570492162 wait complete. 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.876+0000 7f889a416640 10 monclient: build_initial_monmap 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.876+0000 7f889a416640 10 monclient: monmap: 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:40.859 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.876+0000 7f889a416640 1 librados: starting msgr at 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.876+0000 7f889a416640 1 librados: starting objecter 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.876+0000 7f889a416640 1 Processor -- start 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.877+0000 7f889a416640 1 -- start start 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.877+0000 7f889a416640 1 librados: setting wanted keys 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.877+0000 7f889a416640 1 librados: calling monclient init 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.877+0000 7f889a416640 10 monclient: init 2024-03-29T11:13:40.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.878+0000 7f889a416640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.878+0000 7f889a416640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.878+0000 7f889a416640 10 monclient(hunting): picked mon.noname-b con 0x7f88940a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.878+0000 7f889a416640 10 monclient(hunting): picked mon.noname-c con 0x7f889413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 10 monclient(hunting): picked mon.noname-a con 0x7f88941457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88941e19d0 con 0x7f88941457f0 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894142e40 con 0x7f88940a9d70 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894143870 con 0x7f889413f650 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:40.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.879+0000 7f889a416640 10 monclient(hunting): authenticate will time out at 1095.155538s 2024-03-29T11:13:40.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.882+0000 7f8893fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f88940a9d70 0x7f8894135440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:50049/0 (socket says 172.21.15.105:50049) 2024-03-29T11:13:40.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.882+0000 7f8893fff640 1 -- 172.21.15.105:0/2618801376 learned_addr learned my addr 172.21.15.105:0/2618801376 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:40.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.882+0000 7f8890ff9640 10 client.?.objecter ms_handle_connect 0x7f88940a9d70 2024-03-29T11:13:40.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.882+0000 7f8890ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853482737 0 0) 0x7f8894142e40 con 0x7f88940a9d70 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880004eb0 con 0x7f88940a9d70 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 client.?.objecter ms_handle_connect 0x7f88941457f0 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.883+0000 7f8890ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974353350 0 0) 0x7f88941e19d0 con 0x7f88941457f0 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.884+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88800051d0 con 0x7f88941457f0 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.885+0000 7f8890ff9640 10 client.?.objecter ms_handle_connect 0x7f889413f650 2024-03-29T11:13:40.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.885+0000 7f8890ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333234226 0 0) 0x7f88800051d0 con 0x7f88941457f0 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8880005640 con 0x7f88941457f0 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f888c001890 con 0x7f88941457f0 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:40.868 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.888+0000 7f8890ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1277791720 0 0) 0x7f8880005640 con 0x7f88941457f0 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 >> v1:172.21.15.105:6790/0 conn(0x7f889413f650 legacy=0x7f8894142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 >> v1:172.21.15.110:6789/0 conn(0x7f88940a9d70 legacy=0x7f8894135440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:40.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 monclient: found mon.a 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889413e740 con 0x7f88941457f0 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 10 monclient: _renew_subs 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:40.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889413e990 con 0x7f88941457f0 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 10 monclient: _renew_subs 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8894143240 con 0x7f88941457f0 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f889a416640 1 librados: init done 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f888c001d80 con 0x7f88941457f0 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f888c003c10 con 0x7f88941457f0 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:40.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f8890ff9640 10 monclient: dump: 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:40.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.889+0000 7f88927fc640 4 set_mon_vals no callback set 2024-03-29T11:13:40.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.891+0000 7f889a416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.891+0000 7f889a416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.891+0000 7f889a416640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:40.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.891+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8874005080 con 0x7f88941457f0 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.895+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f888c019be0 con 0x7f88941457f0 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.896+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f888c001f70 con 0x7f88941457f0 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.896+0000 7f8890ff9640 10 client.4410.objecter ms_dispatch 0x7f8894000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.896+0000 7f8890ff9640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.896+0000 7f8890ff9640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:40.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.896+0000 7f8890ff9640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:13:40.875 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.897+0000 7f8893fff640 10 monclient: get_auth_request con 0x7f888006e310 auth_method 0 2024-03-29T11:13:40.875 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.897+0000 7f8890ff9640 10 client.4410.objecter ms_handle_connect 0x7f888006e310 2024-03-29T11:13:40.886 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.908+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f888c04f8f0 con 0x7f88941457f0 2024-03-29T11:13:40.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.908+0000 7f8890ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:40.887 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:40.908+0000 7f8890ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:41.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.240+0000 7f889a416640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.240+0000 7f889a416640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.240+0000 7f889a416640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:41.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.240+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8874005330 con 0x7f88941457f0 2024-03-29T11:13:41.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.243+0000 7f8890ff9640 1 -- 172.21.15.105:0/2618801376 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f888c04f8f0 con 0x7f88941457f0 2024-03-29T11:13:41.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.243+0000 7f8890ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:41.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.243+0000 7f8890ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:41.225 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:41.225 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:41.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.250+0000 7f889a416640 10 librados: watch_flush enter 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.250+0000 7f889a416640 10 librados: watch_flush exit 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.250+0000 7f889a416640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.251+0000 7f889a416640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.251+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 >> v1:172.21.15.110:6816/1087970004 conn(0x7f888006e310 legacy=0x7f88800707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.251+0000 7f889a416640 10 monclient: shutdown 2024-03-29T11:13:41.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.251+0000 7f889a416640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:41.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.251+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 >> v1:172.21.15.105:6789/0 conn(0x7f88941457f0 legacy=0x7f88941e3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.252+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 shutdown_connections 2024-03-29T11:13:41.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.252+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 >> 172.21.15.105:0/2618801376 conn(0x7f8894066a00 msgr2=0x7f88940b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:41.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.254+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 shutdown_connections 2024-03-29T11:13:41.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.256+0000 7f889a416640 1 -- 172.21.15.105:0/2618801376 wait complete. 2024-03-29T11:13:41.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:41.256+0000 7f889a416640 1 librados: shutdown 2024-03-29T11:13:41.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:48.064 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.247+0000 7fdfcb094640 10 monclient: get_monmap_and_config 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.247+0000 7fdfcb094640 10 monclient: build_initial_monmap 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.247+0000 7fdfcb094640 10 monclient: monmap: 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:48.248291+0000 2024-03-29T11:13:48.225 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:48.248291+0000 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.248+0000 7fdfcb094640 1 Processor -- start 2024-03-29T11:13:48.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.248+0000 7fdfcb094640 1 -- start start 2024-03-29T11:13:48.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.248+0000 7fdfcb094640 10 monclient: init 2024-03-29T11:13:48.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:48.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:48.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc40b2800 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc40a9c00 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc40b38c0 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4141eb0 con 0x7fdfc40b2800 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40b22d0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4142d30 con 0x7fdfc40b38c0 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:48.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.249+0000 7fdfcb094640 10 monclient(hunting): authenticate will time out at 1102.525635s 2024-03-29T11:13:48.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfca092640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fdfc40b2800 0x7fdfc4142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:54103/0 (socket says 172.21.15.105:54103) 2024-03-29T11:13:48.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfca092640 1 -- 172.21.15.105:0/3320202212 learned_addr learned my addr 172.21.15.105:0/3320202212 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212931901 0 0) 0x7fdfc40b22d0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.256+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfac003520 con 0x7fdfc40a9c00 2024-03-29T11:13:48.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283217057 0 0) 0x7fdfac003520 con 0x7fdfc40a9c00 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 >> v1:172.21.15.105:6790/0 conn(0x7fdfc40b38c0 legacy=0x7fdfc414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 >> v1:172.21.15.105:6789/0 conn(0x7fdfc40b2800 legacy=0x7fdfc4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: _finish_hunting 0 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 monclient: found mon.noname-c 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40aaf90 con 0x7fdfc40a9c00 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: _finish_auth 0 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: _check_auth_tickets 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfb8001690 con 0x7fdfc40a9c00 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: dump: 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.237 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfcb094640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfcb094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfb80017f0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 1 -- 172.21.15.105:0/3320202212 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfb8001c00 con 0x7fdfc40a9c00 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfc9090640 10 monclient: dump: 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.238 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfcb094640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:48.239 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.257+0000 7fdfcb094640 4 set_mon_vals no callback set 2024-03-29T11:13:48.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.258+0000 7fdfcb094640 10 monclient: shutdown 2024-03-29T11:13:48.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.258+0000 7fdfcb094640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:48.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.258+0000 7fdfcb094640 1 -- 172.21.15.105:0/3320202212 >> v1:172.21.15.110:6789/0 conn(0x7fdfc40a9c00 legacy=0x7fdfc413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.241 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.259+0000 7fdfcb094640 1 -- 172.21.15.105:0/3320202212 shutdown_connections 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.259+0000 7fdfcb094640 1 -- 172.21.15.105:0/3320202212 >> 172.21.15.105:0/3320202212 conn(0x7fdfc4066a00 msgr2=0x7fdfc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.259+0000 7fdfcb094640 1 -- 172.21.15.105:0/3320202212 shutdown_connections 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.259+0000 7fdfcb094640 1 -- 172.21.15.105:0/3320202212 wait complete. 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.260+0000 7fdfcb094640 10 monclient: build_initial_monmap 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.260+0000 7fdfcb094640 10 monclient: monmap: 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:48.242 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.260+0000 7fdfcb094640 1 librados: starting msgr at 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.260+0000 7fdfcb094640 1 librados: starting objecter 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.260+0000 7fdfcb094640 1 Processor -- start 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.261+0000 7fdfcb094640 1 -- start start 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.261+0000 7fdfcb094640 1 librados: setting wanted keys 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.261+0000 7fdfcb094640 1 librados: calling monclient init 2024-03-29T11:13:48.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.261+0000 7fdfcb094640 10 monclient: init 2024-03-29T11:13:48.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.262+0000 7fdfcb094640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:48.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.262+0000 7fdfcb094640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:48.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.266+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc40a9c00 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:48.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.266+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc40b2800 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.266+0000 7fdfc9891640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fdfc40b2800 0x7fdfc40706c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:40290/0 (socket says 172.21.15.105:40290) 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.266+0000 7fdfca092640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fdfc40a9c00 0x7fdfc413ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:54233/0 (socket says 172.21.15.105:54233) 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc40b38c0 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4071af0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4071dd0 con 0x7fdfc40b38c0 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc407d340 con 0x7fdfc40b2800 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:48.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfcb094640 10 monclient(hunting): authenticate will time out at 1102.543199s 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.267+0000 7fdfc9891640 1 -- 172.21.15.105:0/1277471017 learned_addr learned my addr 172.21.15.105:0/1277471017 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 client.?.objecter ms_handle_connect 0x7fdfc40b2800 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 client.?.objecter ms_handle_connect 0x7fdfc40a9c00 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275674185 0 0) 0x7fdfc407d340 con 0x7fdfc40b2800 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:13:48.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.268+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb0004eb0 con 0x7fdfc40b2800 2024-03-29T11:13:48.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898930041 0 0) 0x7fdfc4071af0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc407d340 con 0x7fdfc40a9c00 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763752330 0 0) 0x7fdfb0004eb0 con 0x7fdfc40b2800 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc4071af0 con 0x7fdfc40b2800 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfb8001bc0 con 0x7fdfc40b2800 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.248 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.269+0000 7fdfc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 client.?.objecter ms_handle_connect 0x7fdfc40b38c0 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463100032 0 0) 0x7fdfc407d340 con 0x7fdfc40a9c00 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb0004eb0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfbc001890 con 0x7fdfc40a9c00 2024-03-29T11:13:48.249 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.250 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612239261 0 0) 0x7fdfc4071dd0 con 0x7fdfc40b38c0 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.270+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc407d340 con 0x7fdfc40b38c0 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1082725709 0 0) 0x7fdfc407d340 con 0x7fdfc40b38c0 2024-03-29T11:13:48.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc4071dd0 con 0x7fdfc40b38c0 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfb4001750 con 0x7fdfc40b38c0 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 91194039 0 0) 0x7fdfb0004eb0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 >> v1:172.21.15.105:6790/0 conn(0x7fdfc40b2800 legacy=0x7fdfc40706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 >> v1:172.21.15.110:6789/0 conn(0x7fdfc40b38c0 legacy=0x7fdfc41287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 monclient: found mon.a 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc407d4d0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.271+0000 7fdfc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 10 monclient: _renew_subs 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc408bbf0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 10 monclient: _renew_subs 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc4070fb0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfcb094640 1 librados: init done 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfbc001d40 con 0x7fdfc40a9c00 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc888f640 4 set_mon_vals no callback set 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdfbc002090 con 0x7fdfc40a9c00 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.272+0000 7fdfc2ffd640 10 monclient: dump: 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:48.257 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.274+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fdfbc01ac80 con 0x7fdfc40a9c00 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.275+0000 7fdfc9891640 10 monclient: get_auth_request con 0x7fdfb006e160 auth_method 0 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.275+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fdfbc084cd0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.275+0000 7fdfc2ffd640 10 client.4413.objecter ms_dispatch 0x7fdfc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.275+0000 7fdfc2ffd640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.275+0000 7fdfc2ffd640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.276+0000 7fdfc2ffd640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:13:48.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.276+0000 7fdfc2ffd640 10 client.4413.objecter ms_handle_connect 0x7fdfb006e160 2024-03-29T11:13:48.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.277+0000 7fdfcb094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.277+0000 7fdfcb094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.277+0000 7fdfcb094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:48.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.277+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa4005080 con 0x7fdfc40a9c00 2024-03-29T11:13:48.265 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.288+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfbc0503e0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.265 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.288+0000 7fdfc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:48.265 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.288+0000 7fdfc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:48.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.646+0000 7fdfcb094640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.646+0000 7fdfcb094640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.646+0000 7fdfcb094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:13:48.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.646+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfa4005330 con 0x7fdfc40a9c00 2024-03-29T11:13:48.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.651+0000 7fdfc2ffd640 1 -- 172.21.15.105:0/1277471017 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fdfbc0524e0 con 0x7fdfc40a9c00 2024-03-29T11:13:48.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.651+0000 7fdfc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:48.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.651+0000 7fdfc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:48.632 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:48.632 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:48.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 10 librados: watch_flush enter 2024-03-29T11:13:48.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 10 librados: watch_flush exit 2024-03-29T11:13:48.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:13:48.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 >> v1:172.21.15.110:6816/1087970004 conn(0x7fdfb006e160 legacy=0x7fdfb0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 10 monclient: shutdown 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 >> v1:172.21.15.105:6789/0 conn(0x7fdfc40a9c00 legacy=0x7fdfc413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 shutdown_connections 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.657+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 >> 172.21.15.105:0/1277471017 conn(0x7fdfc4066a00 msgr2=0x7fdfc40b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.658+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 shutdown_connections 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.658+0000 7fdfcb094640 1 -- 172.21.15.105:0/1277471017 wait complete. 2024-03-29T11:13:48.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:48.658+0000 7fdfcb094640 1 librados: shutdown 2024-03-29T11:13:48.654 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:55.462 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.633+0000 7fb977fc6640 10 monclient: get_monmap_and_config 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.633+0000 7fb977fc6640 10 monclient: build_initial_monmap 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.633+0000 7fb977fc6640 10 monclient: monmap: 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:13:55.634077+0000 2024-03-29T11:13:55.611 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:13:55.634077+0000 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:13:55.612 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.636+0000 7fb977fc6640 1 Processor -- start 2024-03-29T11:13:55.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.637+0000 7fb977fc6640 1 -- start start 2024-03-29T11:13:55.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.637+0000 7fb977fc6640 10 monclient: init 2024-03-29T11:13:55.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:55.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb970071c30 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb9700a9750 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb97013e1e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700ab300 con 0x7fb97013e1e0 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700ac2f0 con 0x7fb9700a9750 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700ad8d0 con 0x7fb970071c30 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:55.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.638+0000 7fb977fc6640 10 monclient(hunting): authenticate will time out at 1109.914398s 2024-03-29T11:13:55.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.642+0000 7fb975561640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb9700a9750 0x7fb97013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:52475/0 (socket says 172.21.15.105:52475) 2024-03-29T11:13:55.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.642+0000 7fb975561640 1 -- 172.21.15.105:0/2478477501 learned_addr learned my addr 172.21.15.105:0/2478477501 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:55.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648887135 0 0) 0x7fb9700ad8d0 con 0x7fb970071c30 2024-03-29T11:13:55.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c003520 con 0x7fb970071c30 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863771175 0 0) 0x7fb9700ab300 con 0x7fb97013e1e0 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.643+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9700ad8d0 con 0x7fb97013e1e0 2024-03-29T11:13:55.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041561893 0 0) 0x7fb9700ac2f0 con 0x7fb9700a9750 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9700ab300 con 0x7fb9700a9750 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344630985 0 0) 0x7fb95c003520 con 0x7fb970071c30 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 >> v1:172.21.15.110:6789/0 conn(0x7fb9700a9750 legacy=0x7fb97013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 >> v1:172.21.15.105:6789/0 conn(0x7fb97013e1e0 legacy=0x7fb9700ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient: _finish_hunting 0 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 monclient: found mon.noname-b 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:55.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.644+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700ada60 con 0x7fb970071c30 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: _finish_auth 0 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: _check_auth_tickets 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb960001690 con 0x7fb970071c30 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: dump: 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.629 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9600018b0 con 0x7fb970071c30 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 1 -- 172.21.15.105:0/2478477501 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb960001c40 con 0x7fb970071c30 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb974d60640 10 monclient: dump: 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:55.630 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:55.631 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb977fc6640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb977fc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.645+0000 7fb977fc6640 4 set_mon_vals no callback set 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.646+0000 7fb977fc6640 10 monclient: shutdown 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.646+0000 7fb977fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.646+0000 7fb977fc6640 1 -- 172.21.15.105:0/2478477501 >> v1:172.21.15.105:6790/0 conn(0x7fb970071c30 legacy=0x7fb9700b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.647+0000 7fb977fc6640 1 -- 172.21.15.105:0/2478477501 shutdown_connections 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.647+0000 7fb977fc6640 1 -- 172.21.15.105:0/2478477501 >> 172.21.15.105:0/2478477501 conn(0x7fb97006b660 msgr2=0x7fb97006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:55.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.648+0000 7fb977fc6640 1 -- 172.21.15.105:0/2478477501 shutdown_connections 2024-03-29T11:13:55.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.648+0000 7fb977fc6640 1 -- 172.21.15.105:0/2478477501 wait complete. 2024-03-29T11:13:55.644 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 10 monclient: build_initial_monmap 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 10 monclient: monmap: 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 librados: starting msgr at 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 librados: starting objecter 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 Processor -- start 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 -- start start 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 librados: setting wanted keys 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 1 librados: calling monclient init 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.649+0000 7fb977fc6640 10 monclient: init 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb977fc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb977fc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb970071c30 addr v1:172.21.15.105:6790/0 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb975d62640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fb970071c30 0x7fb97011d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55708/0 (socket says 172.21.15.105:55708) 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb975d62640 1 -- 172.21.15.105:0/2448544891 learned_addr learned my addr 172.21.15.105:0/2448544891 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:13:55.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.650+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb9700a9750 addr v1:172.21.15.110:6789/0 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb97013e1e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97011c820 con 0x7fb97013e1e0 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97011cb00 con 0x7fb9700a9750 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97011cde0 con 0x7fb970071c30 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb977fc6640 10 monclient(hunting): authenticate will time out at 1109.929475s 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb9667fc640 10 client.?.objecter ms_handle_connect 0x7fb970071c30 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb9667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532719960 0 0) 0x7fb97011cde0 con 0x7fb970071c30 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.653+0000 7fb9667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:13:55.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb954004eb0 con 0x7fb970071c30 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 client.?.objecter ms_handle_connect 0x7fb9700a9750 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879368646 0 0) 0x7fb97011cb00 con 0x7fb9700a9750 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb97011cde0 con 0x7fb9700a9750 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.654+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4095159148 0 0) 0x7fb954004eb0 con 0x7fb970071c30 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb97011cb00 con 0x7fb970071c30 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb960001690 con 0x7fb970071c30 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 10 monclient(hunting): dump: 2024-03-29T11:13:55.648 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888034363 0 0) 0x7fb97011cb00 con 0x7fb970071c30 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 >> v1:172.21.15.110:6789/0 conn(0x7fb9700a9750 legacy=0x7fb97011b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.655+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 >> v1:172.21.15.105:6789/0 conn(0x7fb97013e1e0 legacy=0x7fb9701e6800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:55.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 1 monclient: found mon.c 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700b2fa0 con 0x7fb970071c30 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb977fc6640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb977fc6640 10 monclient: _renew_subs 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb977fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.657+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9700b3190 con 0x7fb970071c30 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb977fc6640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb977fc6640 10 monclient: _renew_subs 2024-03-29T11:13:55.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb977fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97012c380 con 0x7fb970071c30 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb977fc6640 1 librados: init done 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9600038a0 con 0x7fb970071c30 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb960003cb0 con 0x7fb970071c30 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9667fc640 10 monclient: dump: 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:55.651 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.658+0000 7fb9677fe640 4 set_mon_vals no callback set 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.660+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb960003e90 con 0x7fb970071c30 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.660+0000 7fb977fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.660+0000 7fb977fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.660+0000 7fb977fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.660+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb958005080 con 0x7fb970071c30 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fb960084bc0 con 0x7fb970071c30 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb9667fc640 10 client.4472.objecter ms_dispatch 0x7fb970000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb975561640 10 monclient: get_auth_request con 0x7fb95406dd80 auth_method 0 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb9667fc640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:55.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb9667fc640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:55.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.661+0000 7fb9667fc640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:13:55.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.662+0000 7fb9667fc640 10 client.4472.objecter ms_handle_connect 0x7fb95406dd80 2024-03-29T11:13:55.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.671+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9600504a0 con 0x7fb970071c30 2024-03-29T11:13:55.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.671+0000 7fb9667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:55.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:55.671+0000 7fb9667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:55.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.010+0000 7fb977fc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.010+0000 7fb977fc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.010+0000 7fb977fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:13:55.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.010+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb958005330 con 0x7fb970071c30 2024-03-29T11:13:55.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.012+0000 7fb9667fc640 1 -- 172.21.15.105:0/2448544891 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fb9600523d0 con 0x7fb970071c30 2024-03-29T11:13:55.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.012+0000 7fb9667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:55.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.012+0000 7fb9667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:55.993 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:13:55.993 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 10 librados: watch_flush enter 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 10 librados: watch_flush exit 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb95406dd80 legacy=0x7fb954070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 10 monclient: shutdown 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 >> v1:172.21.15.105:6790/0 conn(0x7fb970071c30 legacy=0x7fb97011d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 shutdown_connections 2024-03-29T11:13:55.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.021+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 >> 172.21.15.105:0/2448544891 conn(0x7fb97006b660 msgr2=0x7fb97013d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:56.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.022+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 shutdown_connections 2024-03-29T11:13:56.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.022+0000 7fb977fc6640 1 -- 172.21.15.105:0/2448544891 wait complete. 2024-03-29T11:13:56.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:13:56.025+0000 7fb977fc6640 1 librados: shutdown 2024-03-29T11:13:56.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:02.830 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.003+0000 7f34aaf89640 10 monclient: get_monmap_and_config 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.003+0000 7f34aaf89640 10 monclient: build_initial_monmap 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.003+0000 7f34aaf89640 10 monclient: monmap: 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:02.981 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:03.004471+0000 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:03.004471+0000 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:02.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.004+0000 7f34aaf89640 1 Processor -- start 2024-03-29T11:14:02.983 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.004+0000 7f34aaf89640 1 -- start start 2024-03-29T11:14:02.983 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.004+0000 7f34aaf89640 10 monclient: init 2024-03-29T11:14:02.983 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-b con 0x7f34a41457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-c con 0x7f34a413f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-a con 0x7f34a40a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40aaef0 con 0x7f34a40a9d70 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34a3fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f34a413f650 0x7f34a40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:52488/0 (socket says 172.21.15.105:52488) 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34a3fff640 1 -- 172.21.15.105:0/2490708777 learned_addr learned my addr 172.21.15.105:0/2490708777 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.005+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a4145bd0 con 0x7f34a413f650 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a4145eb0 con 0x7f34a41457f0 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34aaf89640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:02.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34aaf89640 10 monclient(hunting): authenticate will time out at 1117.282169s 2024-03-29T11:14:02.985 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98227103 0 0) 0x7f34a4145eb0 con 0x7f34a41457f0 2024-03-29T11:14:02.985 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:02.985 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:02.985 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:02.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:14:02.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494003520 con 0x7f34a41457f0 2024-03-29T11:14:02.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499408714 0 0) 0x7f34a40aaef0 con 0x7f34a40a9d70 2024-03-29T11:14:02.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:02.986 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:02.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.006+0000 7f34a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:02.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:14:02.989 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a4145eb0 con 0x7f34a40a9d70 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117561035 0 0) 0x7f3494003520 con 0x7f34a41457f0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 >> v1:172.21.15.110:6789/0 conn(0x7f34a413f650 legacy=0x7f34a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 >> v1:172.21.15.105:6789/0 conn(0x7f34a40a9d70 legacy=0x7f34a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 monclient: found mon.noname-b 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4146340 con 0x7f34a41457f0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3498001690 con 0x7f34a41457f0 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:02.990 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: dump: 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:02.991 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34aaf89640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34aaf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34980017f0 con 0x7f34a41457f0 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 1 -- 172.21.15.105:0/2490708777 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3498001c00 con 0x7f34a41457f0 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34a37fe640 10 monclient: dump: 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:02.992 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34aaf89640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:02.993 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.007+0000 7f34aaf89640 4 set_mon_vals no callback set 2024-03-29T11:14:02.998 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 10 monclient: shutdown 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 >> v1:172.21.15.105:6790/0 conn(0x7f34a41457f0 legacy=0x7f34a413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 shutdown_connections 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 >> 172.21.15.105:0/2490708777 conn(0x7f34a4066a00 msgr2=0x7f34a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 shutdown_connections 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.009+0000 7f34aaf89640 1 -- 172.21.15.105:0/2490708777 wait complete. 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.010+0000 7f34aaf89640 10 monclient: build_initial_monmap 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.010+0000 7f34aaf89640 10 monclient: monmap: 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:02.999 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.010+0000 7f34aaf89640 1 librados: starting msgr at 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.010+0000 7f34aaf89640 1 librados: starting objecter 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.010+0000 7f34aaf89640 1 Processor -- start 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.011+0000 7f34aaf89640 1 -- start start 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.011+0000 7f34aaf89640 1 librados: setting wanted keys 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.011+0000 7f34aaf89640 1 librados: calling monclient init 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.011+0000 7f34aaf89640 10 monclient: init 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34aaf89640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34aaf89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:03.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-c con 0x7f34a40a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34a8d25640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f34a40a9d70 0x7f34a413b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:49780/0 (socket says 172.21.15.105:49780) 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34a8d25640 1 -- 172.21.15.105:0/1920364897 learned_addr learned my addr 172.21.15.105:0/1920364897 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-a con 0x7f34a413f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34a17fa640 10 client.?.objecter ms_handle_connect 0x7f34a40a9d70 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.012+0000 7f34aaf89640 10 monclient(hunting): picked mon.noname-b con 0x7f34a41457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 client.?.objecter ms_handle_connect 0x7f34a413f650 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a41ccc10 con 0x7f34a413f650 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a41ccef0 con 0x7f34a41457f0 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a41cb6b0 con 0x7f34a40a9d70 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34aaf89640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:03.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34aaf89640 10 monclient(hunting): authenticate will time out at 1117.289115s 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 client.?.objecter ms_handle_connect 0x7f34a41457f0 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162154098 0 0) 0x7f34a41ccef0 con 0x7f34a41457f0 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3488004eb0 con 0x7f34a41457f0 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359285494 0 0) 0x7f34a41ccc10 con 0x7f34a413f650 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.002 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a41ccef0 con 0x7f34a413f650 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 276568068 0 0) 0x7f3488004eb0 con 0x7f34a41457f0 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34a41ccc10 con 0x7f34a41457f0 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f349c001cb0 con 0x7f34a41457f0 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): dump: 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.003 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.013+0000 7f34a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021204126 0 0) 0x7f34a41cb6b0 con 0x7f34a40a9d70 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3488004eb0 con 0x7f34a40a9d70 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596240496 0 0) 0x7f3488004eb0 con 0x7f34a40a9d70 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34a41cb6b0 con 0x7f34a40a9d70 2024-03-29T11:14:03.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f3498001bc0 con 0x7f34a40a9d70 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient(hunting): dump: 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:03.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2904901503 0 0) 0x7f34a41ccc10 con 0x7f34a41457f0 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 >> v1:172.21.15.105:6790/0 conn(0x7f34a40a9d70 legacy=0x7f34a413b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 >> v1:172.21.15.105:6789/0 conn(0x7f34a413f650 legacy=0x7f34a4138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 monclient: found mon.b 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a41cb340 con 0x7f34a41457f0 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 10 monclient: _renew_subs 2024-03-29T11:14:03.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a41393d0 con 0x7f34a41457f0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 10 monclient: _renew_subs 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a41cbdb0 con 0x7f34a41457f0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.014+0000 7f34aaf89640 1 librados: init done 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f349c003990 con 0x7f34a41457f0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f349c003da0 con 0x7f34a41457f0 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a17fa640 10 monclient: dump: 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:03.007 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.015+0000 7f34a2ffd640 4 set_mon_vals no callback set 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.016+0000 7f34aaf89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.016+0000 7f34aaf89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.016+0000 7f34aaf89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.016+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484005080 con 0x7f34a41457f0 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.017+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f349c003f00 con 0x7f34a41457f0 2024-03-29T11:14:03.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f349c083b60 con 0x7f34a41457f0 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 10 client.4429.objecter ms_dispatch 0x7f34a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a8d25640 10 monclient: get_auth_request con 0x7f348806de50 auth_method 0 2024-03-29T11:14:03.009 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.018+0000 7f34a17fa640 10 client.4429.objecter ms_handle_connect 0x7f348806de50 2024-03-29T11:14:03.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.026+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f349c04f440 con 0x7f34a41457f0 2024-03-29T11:14:03.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.026+0000 7f34a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.010 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.026+0000 7f34a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:03.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.402+0000 7f34aaf89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.402+0000 7f34aaf89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.402+0000 7f34aaf89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:03.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.402+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3484005330 con 0x7f34a41457f0 2024-03-29T11:14:03.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.403+0000 7f34a17fa640 1 -- 172.21.15.105:0/1920364897 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f349c051370 con 0x7f34a41457f0 2024-03-29T11:14:03.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.403+0000 7f34a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:03.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.403+0000 7f34a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:03.381 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:03.382 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 10 librados: watch_flush enter 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 10 librados: watch_flush exit 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 10 client.4429.objecter successfully canceled tick 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 >> v1:172.21.15.110:6816/1087970004 conn(0x7f348806de50 legacy=0x7f3488070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 10 monclient: shutdown 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.407+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 >> v1:172.21.15.110:6789/0 conn(0x7f34a41457f0 legacy=0x7f34a41ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.408+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 shutdown_connections 2024-03-29T11:14:03.387 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.408+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 >> 172.21.15.105:0/1920364897 conn(0x7f34a4066a00 msgr2=0x7f34a413cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:03.387 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.408+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 shutdown_connections 2024-03-29T11:14:03.387 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.408+0000 7f34aaf89640 1 -- 172.21.15.105:0/1920364897 wait complete. 2024-03-29T11:14:03.387 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:03.408+0000 7f34aaf89640 1 librados: shutdown 2024-03-29T11:14:03.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:10.208 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:10.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.363+0000 7f894f33d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:10.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.363+0000 7f894f33d640 10 monclient: build_initial_monmap 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.363+0000 7f894f33d640 10 monclient: monmap: 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:10.364725+0000 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:10.364725+0000 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:10.342 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.364+0000 7f894f33d640 1 Processor -- start 2024-03-29T11:14:10.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.364+0000 7f894f33d640 1 -- start start 2024-03-29T11:14:10.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.364+0000 7f894f33d640 10 monclient: init 2024-03-29T11:14:10.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-b con 0x7f89481457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-a con 0x7f894813f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-c con 0x7f89480a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89480aaef0 con 0x7f894813f650 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8948145bd0 con 0x7f89480a9d70 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8948145eb0 con 0x7f89481457f0 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894f33d640 10 monclient(hunting): authenticate will time out at 1124.641561s 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894d0d9640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f89481457f0 0x7f894813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41256/0 (socket says 172.21.15.105:41256) 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.365+0000 7f894d0d9640 1 -- 172.21.15.105:0/2055981075 learned_addr learned my addr 172.21.15.105:0/2055981075 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529076333 0 0) 0x7f8948145eb0 con 0x7f89481457f0 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934003520 con 0x7f89481457f0 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4240723963 0 0) 0x7f8934003520 con 0x7f89481457f0 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 >> v1:172.21.15.110:6789/0 conn(0x7f89480a9d70 legacy=0x7f894814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:10.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.366+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 >> v1:172.21.15.105:6789/0 conn(0x7f894813f650 legacy=0x7f89480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.368+0000 7f893ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 1 monclient: found mon.noname-b 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948146340 con 0x7f89481457f0 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: _finish_auth 0 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:10.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8944001690 con 0x7f89481457f0 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: dump: 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:10.349 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f894f33d640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f894f33d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8944001920 con 0x7f89481457f0 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 1 -- 172.21.15.105:0/2055981075 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8944001cb0 con 0x7f89481457f0 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f894f33d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f894f33d640 4 set_mon_vals no callback set 2024-03-29T11:14:10.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.369+0000 7f893ffff640 10 monclient: dump: 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 10 monclient: shutdown 2024-03-29T11:14:10.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:10.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 1 -- 172.21.15.105:0/2055981075 >> v1:172.21.15.105:6790/0 conn(0x7f89481457f0 legacy=0x7f894813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 1 -- 172.21.15.105:0/2055981075 shutdown_connections 2024-03-29T11:14:10.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 1 -- 172.21.15.105:0/2055981075 >> 172.21.15.105:0/2055981075 conn(0x7f8948066a00 msgr2=0x7f8948066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:10.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 1 -- 172.21.15.105:0/2055981075 shutdown_connections 2024-03-29T11:14:10.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.370+0000 7f894f33d640 1 -- 172.21.15.105:0/2055981075 wait complete. 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.371+0000 7f894f33d640 10 monclient: build_initial_monmap 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.371+0000 7f894f33d640 10 monclient: monmap: 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:10.354 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.371+0000 7f894f33d640 1 librados: starting msgr at 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.371+0000 7f894f33d640 1 librados: starting objecter 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.372+0000 7f894f33d640 1 Processor -- start 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.372+0000 7f894f33d640 1 -- start start 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.372+0000 7f894f33d640 1 librados: setting wanted keys 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.372+0000 7f894f33d640 1 librados: calling monclient init 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.372+0000 7f894f33d640 10 monclient: init 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:10.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-b con 0x7f89480a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-c con 0x7f894813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient(hunting): picked mon.noname-a con 0x7f89481457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8948140350 con 0x7f89481457f0 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8948141ac0 con 0x7f89480a9d70 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89481357e0 con 0x7f894813f650 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.373+0000 7f894f33d640 10 monclient(hunting): authenticate will time out at 1124.649779s 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f894c8d8640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f894813f650 0x7f8948143320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:42182/0 (socket says 172.21.15.105:42182) 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f894c8d8640 1 -- 172.21.15.105:0/195539412 learned_addr learned my addr 172.21.15.105:0/195539412 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 client.?.objecter ms_handle_connect 0x7f894813f650 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308896830 0 0) 0x7f89481357e0 con 0x7f894813f650 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:10.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c004eb0 con 0x7f894813f650 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204062242 0 0) 0x7f892c004eb0 con 0x7f894813f650 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89481357e0 con 0x7f894813f650 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8938001960 con 0x7f894813f650 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): dump: 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.357 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.374+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 176524192 0 0) 0x7f89481357e0 con 0x7f894813f650 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.375+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 >> v1:172.21.15.110:6789/0 conn(0x7f89480a9d70 legacy=0x7f89481350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.358 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.375+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 >> v1:172.21.15.105:6789/0 conn(0x7f89481457f0 legacy=0x7f89481e27d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 1 monclient: found mon.c 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948135970 con 0x7f894813f650 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:10.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.377+0000 7f893dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8938001ac0 con 0x7f894813f650 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 10 monclient: _renew_subs 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894813fda0 con 0x7f894813f650 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8938003a40 con 0x7f894813f650 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f893dffb640 10 monclient: dump: 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.363 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 10 monclient: _renew_subs 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8948145140 con 0x7f894813f650 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.378+0000 7f894f33d640 1 librados: init done 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.380+0000 7f894f33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.380+0000 7f894f33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.380+0000 7f894f33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.380+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948075f10 con 0x7f894813f650 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.380+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8938003ba0 con 0x7f894813f650 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893effd640 4 set_mon_vals no callback set 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8938084b40 con 0x7f894813f650 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893dffb640 10 client.4484.objecter ms_dispatch 0x7f8948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893dffb640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893dffb640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f893dffb640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.381+0000 7f894d0d9640 10 monclient: get_auth_request con 0x7f892c06ddb0 auth_method 0 2024-03-29T11:14:10.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.382+0000 7f893dffb640 10 client.4484.objecter ms_handle_connect 0x7f892c06ddb0 2024-03-29T11:14:10.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.388+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8938050420 con 0x7f894813f650 2024-03-29T11:14:10.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.388+0000 7f893dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:10.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.388+0000 7f893dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.685+0000 7f894f33d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.685+0000 7f894f33d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.685+0000 7f894f33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.685+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8948140500 con 0x7f894813f650 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.686+0000 7f893dffb640 1 -- 172.21.15.105:0/195539412 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8948140500 con 0x7f894813f650 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.686+0000 7f893dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:10.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.686+0000 7f893dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:10.665 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:10.665 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:10.667 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 10 librados: watch_flush enter 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 10 librados: watch_flush exit 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 >> v1:172.21.15.110:6816/1087970004 conn(0x7f892c06ddb0 legacy=0x7f892c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 10 monclient: shutdown 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 >> v1:172.21.15.105:6790/0 conn(0x7f894813f650 legacy=0x7f8948143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 shutdown_connections 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 >> 172.21.15.105:0/195539412 conn(0x7f8948066a00 msgr2=0x7f89480b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 shutdown_connections 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 -- 172.21.15.105:0/195539412 wait complete. 2024-03-29T11:14:10.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:10.689+0000 7f894f33d640 1 librados: shutdown 2024-03-29T11:14:10.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:17.490 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:17.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.660+0000 7facfbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.660+0000 7facfbfff640 10 monclient: build_initial_monmap 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.660+0000 7facfbfff640 10 monclient: monmap: 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:17.661138+0000 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:17.661138+0000 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.639 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:17.640 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:17.640 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:17.640 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.661+0000 7facfbfff640 1 Processor -- start 2024-03-29T11:14:17.642 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.661+0000 7facfbfff640 1 -- start start 2024-03-29T11:14:17.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.661+0000 7facfbfff640 10 monclient: init 2024-03-29T11:14:17.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.662+0000 7facfbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:17.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.662+0000 7facfbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:17.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.662+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-b con 0x7facfc0a9750 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:17.643 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.662+0000 7facfaffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7facfc0a9750 0x7facfc143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46482/0 (socket says 172.21.15.105:46482) 2024-03-29T11:14:17.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-a con 0x7facfc0adca0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:17.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfaffd640 1 -- 172.21.15.105:0/1984769543 learned_addr learned my addr 172.21.15.105:0/1984769543 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:17.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-c con 0x7facfc146a80 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:17.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0abdc0 con 0x7facfc0adca0 2024-03-29T11:14:17.645 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0b0070 con 0x7facfc146a80 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0af880 con 0x7facfc0a9750 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facfbfff640 10 monclient(hunting): authenticate will time out at 1131.941715s 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.665+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634079860 0 0) 0x7facfc0abdc0 con 0x7facfc0adca0 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.666+0000 7facf9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.666+0000 7facf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.646 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.666+0000 7facf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.666+0000 7facf9ffb640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:14:17.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.666+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4003520 con 0x7facfc0adca0 2024-03-29T11:14:17.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113666265 0 0) 0x7face4003520 con 0x7facfc0adca0 2024-03-29T11:14:17.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 >> v1:172.21.15.105:6790/0 conn(0x7facfc0a9750 legacy=0x7facfc143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 >> v1:172.21.15.110:6789/0 conn(0x7facfc146a80 legacy=0x7facfc151bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:14:17.648 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0ac030 con 0x7facfc0adca0 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7facec0015c0 con 0x7facfc0adca0 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:17.649 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: dump: 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.650 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facfbfff640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facfbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facec001790 con 0x7facfc0adca0 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 1 -- 172.21.15.105:0/1984769543 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7facec001be0 con 0x7facfc0adca0 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.669+0000 7facf9ffb640 10 monclient: dump: 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:17.651 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.670+0000 7facfbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.670+0000 7facfbfff640 4 set_mon_vals no callback set 2024-03-29T11:14:17.652 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.670+0000 7facfbfff640 10 monclient: shutdown 2024-03-29T11:14:17.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.670+0000 7facfbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:17.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.670+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 >> v1:172.21.15.105:6789/0 conn(0x7facfc0adca0 legacy=0x7facfc146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.671+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 shutdown_connections 2024-03-29T11:14:17.653 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.671+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 >> 172.21.15.105:0/1984769543 conn(0x7facfc0673e0 msgr2=0x7facfc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:17.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.672+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 shutdown_connections 2024-03-29T11:14:17.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.672+0000 7facfbfff640 1 -- 172.21.15.105:0/1984769543 wait complete. 2024-03-29T11:14:17.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.673+0000 7facfbfff640 10 monclient: build_initial_monmap 2024-03-29T11:14:17.654 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.673+0000 7facfbfff640 10 monclient: monmap: 2024-03-29T11:14:17.654 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.673+0000 7facfbfff640 1 librados: starting msgr at 2024-03-29T11:14:17.655 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.673+0000 7facfbfff640 1 librados: starting objecter 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.673+0000 7facfbfff640 1 Processor -- start 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 1 -- start start 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 1 librados: setting wanted keys 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 1 librados: calling monclient init 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 10 monclient: init 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:17.656 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.674+0000 7facfbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-b con 0x7facfc0a9750 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-c con 0x7facfc0adca0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 10 monclient(hunting): picked mon.noname-a con 0x7facfc146a80 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0b2460 con 0x7facfc146a80 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0b34b0 con 0x7facfc0a9750 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0b1f40 con 0x7facfc0adca0 2024-03-29T11:14:17.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.677+0000 7facfbfff640 10 monclient(hunting): authenticate will time out at 1131.953323s 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.678+0000 7facfaffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7facfc0a9750 0x7facfc1e72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:44885/0 (socket says 172.21.15.105:44885) 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.678+0000 7facfaffd640 1 -- 172.21.15.105:0/608816286 learned_addr learned my addr 172.21.15.105:0/608816286 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 client.?.objecter ms_handle_connect 0x7facfc0adca0 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 client.?.objecter ms_handle_connect 0x7facfc0a9750 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103291372 0 0) 0x7facfc0b1f40 con 0x7facfc0adca0 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.663 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8004eb0 con 0x7facfc0adca0 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415812097 0 0) 0x7facfc0b34b0 con 0x7facfc0a9750 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.681+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc0b1f40 con 0x7facfc0a9750 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 client.?.objecter ms_handle_connect 0x7facfc146a80 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136181630 0 0) 0x7facfc0b2460 con 0x7facfc146a80 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:17.664 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc0b34b0 con 0x7facfc146a80 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4100988429 0 0) 0x7facfc0b34b0 con 0x7facfc146a80 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facfc0b2460 con 0x7facfc146a80 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7facf0001650 con 0x7facfc146a80 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): dump: 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.665 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.682+0000 7face37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366904561 0 0) 0x7face8004eb0 con 0x7facfc0adca0 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facfc0b34b0 con 0x7facfc0adca0 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7facec001aa0 con 0x7facfc0adca0 2024-03-29T11:14:17.666 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient(hunting): dump: 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.667 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1206587571 0 0) 0x7facfc0b34b0 con 0x7facfc0adca0 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 >> v1:172.21.15.110:6789/0 conn(0x7facfc0a9750 legacy=0x7facfc1e72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 >> v1:172.21.15.105:6789/0 conn(0x7facfc146a80 legacy=0x7facfc1ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 monclient: found mon.c 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:17.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0b20d0 con 0x7facfc0adca0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7face37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7facfbfff640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7facfbfff640 10 monclient: _renew_subs 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7facfbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.683+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc0b14f0 con 0x7facfc0adca0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facfbfff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facfbfff640 10 monclient: _renew_subs 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facfbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc1456b0 con 0x7facfc0adca0 2024-03-29T11:14:17.669 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facfbfff640 1 librados: init done 2024-03-29T11:14:17.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facec001c00 con 0x7facfc0adca0 2024-03-29T11:14:17.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7face37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:17.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.684+0000 7facf97fa640 4 set_mon_vals no callback set 2024-03-29T11:14:17.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.685+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7facec003a40 con 0x7facfc0adca0 2024-03-29T11:14:17.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.685+0000 7face37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:17.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.685+0000 7face37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:17.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.685+0000 7face37fe640 10 monclient: dump: 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.686+0000 7facfbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.686+0000 7facfbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.686+0000 7facfbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.686+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc005080 con 0x7facfc0adca0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.686+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7facec019a10 con 0x7facfc0adca0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.687+0000 7facfaffd640 10 monclient: get_auth_request con 0x7face806dd30 auth_method 0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7facec0839b0 con 0x7facfc0adca0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 10 client.4490.objecter ms_dispatch 0x7facfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:14:17.673 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.688+0000 7face37fe640 10 client.4490.objecter ms_handle_connect 0x7face806dd30 2024-03-29T11:14:17.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.695+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facec04f290 con 0x7facfc0adca0 2024-03-29T11:14:17.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.695+0000 7face37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:17.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:17.695+0000 7face37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:17.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.021+0000 7facfbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:17.999 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.021+0000 7facfbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.021+0000 7facfbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.021+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facdc005330 con 0x7facfc0adca0 2024-03-29T11:14:18.000 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.022+0000 7face37fe640 1 -- 172.21.15.105:0/608816286 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7facec0511c0 con 0x7facfc0adca0 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.022+0000 7face37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.022+0000 7face37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:18.001 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:18.004 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.026+0000 7facfbfff640 10 librados: watch_flush enter 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 10 librados: watch_flush exit 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 >> v1:172.21.15.110:6816/1087970004 conn(0x7face806dd30 legacy=0x7face80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 10 monclient: shutdown 2024-03-29T11:14:18.005 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:18.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 >> v1:172.21.15.105:6790/0 conn(0x7facfc0adca0 legacy=0x7facfc1e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 shutdown_connections 2024-03-29T11:14:18.006 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.027+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 >> 172.21.15.105:0/608816286 conn(0x7facfc0673e0 msgr2=0x7facfc0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:18.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.030+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 shutdown_connections 2024-03-29T11:14:18.007 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.030+0000 7facfbfff640 1 -- 172.21.15.105:0/608816286 wait complete. 2024-03-29T11:14:18.008 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:18.030+0000 7facfbfff640 1 librados: shutdown 2024-03-29T11:14:18.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:24.834 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.004+0000 7fbdaa015640 10 monclient: get_monmap_and_config 2024-03-29T11:14:24.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.004+0000 7fbdaa015640 10 monclient: build_initial_monmap 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.004+0000 7fbdaa015640 10 monclient: monmap: 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:25.005581+0000 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:25.005581+0000 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:24.983 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.011 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.005+0000 7fbdaa015640 1 Processor -- start 2024-03-29T11:14:25.011 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.005+0000 7fbdaa015640 1 -- start start 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.005+0000 7fbdaa015640 10 monclient: init 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.006+0000 7fbdaa015640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.006+0000 7fbdaa015640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.006+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-b con 0x7fbda41457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.006+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-a con 0x7fbda413f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.006+0000 7fbda37fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fbda41457f0 0x7fbda413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55248/0 (socket says 172.21.15.105:55248) 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbda37fe640 1 -- 172.21.15.105:0/39416231 learned_addr learned my addr 172.21.15.105:0/39416231 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-c con 0x7fbda40a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda40aaef0 con 0x7fbda413f650 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4145bd0 con 0x7fbda40a9d70 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4145eb0 con 0x7fbda41457f0 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.007+0000 7fbdaa015640 10 monclient(hunting): authenticate will time out at 1139.283659s 2024-03-29T11:14:25.012 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755937626 0 0) 0x7fbda4145eb0 con 0x7fbda41457f0 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.009+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd8c003520 con 0x7fbda41457f0 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137574813 0 0) 0x7fbda40aaef0 con 0x7fbda413f650 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.010+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda4145eb0 con 0x7fbda413f650 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3417090614 0 0) 0x7fbd8c003520 con 0x7fbda41457f0 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 >> v1:172.21.15.110:6789/0 conn(0x7fbda40a9d70 legacy=0x7fbda414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 >> v1:172.21.15.105:6789/0 conn(0x7fbda413f650 legacy=0x7fbda40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.013 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4146340 con 0x7fbda41457f0 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbd9c001a10 con 0x7fbda41457f0 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:25.014 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbda27fc640 10 monclient: dump: 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbdaa015640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.011+0000 7fbdaa015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd9c001c30 con 0x7fbda41457f0 2024-03-29T11:14:25.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 1 -- 172.21.15.105:0/39416231 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbd9c001fc0 con 0x7fbda41457f0 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbda27fc640 10 monclient: dump: 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbdaa015640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:25.016 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.012+0000 7fbdaa015640 4 set_mon_vals no callback set 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.013+0000 7fbdaa015640 10 monclient: shutdown 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.013+0000 7fbdaa015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.013+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 >> v1:172.21.15.105:6790/0 conn(0x7fbda41457f0 legacy=0x7fbda413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.013+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 shutdown_connections 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.013+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 >> 172.21.15.105:0/39416231 conn(0x7fbda4066a00 msgr2=0x7fbda4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.014+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 shutdown_connections 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.014+0000 7fbdaa015640 1 -- 172.21.15.105:0/39416231 wait complete. 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.015+0000 7fbdaa015640 10 monclient: build_initial_monmap 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.015+0000 7fbdaa015640 10 monclient: monmap: 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:25.017 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.015+0000 7fbdaa015640 1 librados: starting msgr at 2024-03-29T11:14:25.018 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 1 librados: starting objecter 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 1 Processor -- start 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 1 -- start start 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 1 librados: setting wanted keys 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 1 librados: calling monclient init 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.016+0000 7fbdaa015640 10 monclient: init 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.017+0000 7fbdaa015640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.017+0000 7fbdaa015640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.017+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-b con 0x7fbda40a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:25.019 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.017+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-c con 0x7fbda413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.017+0000 7fbdaa015640 10 monclient(hunting): picked mon.noname-a con 0x7fbda41457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbdaa015640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4142890 con 0x7fbda41457f0 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbdaa015640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda4141d60 con 0x7fbda40a9d70 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbdaa015640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda41da9b0 con 0x7fbda413f650 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbdaa015640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbdaa015640 10 monclient(hunting): authenticate will time out at 1139.295354s 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbda2ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fbda413f650 0x7fbda4142180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55624/0 (socket says 172.21.15.105:55624) 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbda2ffd640 1 -- 172.21.15.105:0/2890500854 learned_addr learned my addr 172.21.15.105:0/2890500854 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbd8bfff640 10 client.?.objecter ms_handle_connect 0x7fbda413f650 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbd8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615182334 0 0) 0x7fbda41da9b0 con 0x7fbda413f650 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbd8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.019+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.020+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.020 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.020+0000 7fbd8bfff640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.020+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd90004eb0 con 0x7fbda413f650 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 client.?.objecter ms_handle_connect 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773123872 0 0) 0x7fbd98001270 con 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda41da9b0 con 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675355353 0 0) 0x7fbda41da9b0 con 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd98001270 con 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbd980018e0 con 0x7fbda41457f0 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): dump: 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3455594548 0 0) 0x7fbd98001270 con 0x7fbda41457f0 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 >> v1:172.21.15.105:6790/0 conn(0x7fbda413f650 legacy=0x7fbda4142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 >> v1:172.21.15.110:6789/0 conn(0x7fbda40a9d70 legacy=0x7fbda413fda0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 monclient: found mon.a 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda41d4ae0 con 0x7fbda41457f0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbd8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 10 monclient: _renew_subs 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda413b3f0 con 0x7fbda41457f0 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:25.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 10 monclient: _renew_subs 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.021+0000 7fbdaa015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.022+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda41da6b0 con 0x7fbda41457f0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.022+0000 7fbdaa015640 1 librados: init done 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbdaa015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbdaa015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbdaa015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda4075f10 con 0x7fbda41457f0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd98001d30 con 0x7fbda41457f0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbd980037b0 con 0x7fbda41457f0 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbd8bfff640 10 monclient: dump: 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:25.024 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.023+0000 7fbda1ffb640 4 set_mon_vals no callback set 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.024+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fbd98019800 con 0x7fbda41457f0 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.025+0000 7fbda37fe640 10 monclient: get_auth_request con 0x7fbd9006dc90 auth_method 0 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.026+0000 7fbd8bfff640 10 client.4434.objecter ms_handle_connect 0x7fbd9006dc90 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.028+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fbd98053560 con 0x7fbda41457f0 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.028+0000 7fbd8bfff640 10 client.4434.objecter ms_dispatch 0x7fbda4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:25.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.028+0000 7fbd8bfff640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:25.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.028+0000 7fbd8bfff640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:25.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.028+0000 7fbd8bfff640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:14:25.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.035+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd9804eea0 con 0x7fbda41457f0 2024-03-29T11:14:25.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.035+0000 7fbd8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.035+0000 7fbd8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:25.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.371+0000 7fbdaa015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.371+0000 7fbdaa015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.371+0000 7fbdaa015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:25.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.371+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbda40b3680 con 0x7fbda41457f0 2024-03-29T11:14:25.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.374+0000 7fbd8bfff640 1 -- 172.21.15.105:0/2890500854 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fbd98050fa0 con 0x7fbda41457f0 2024-03-29T11:14:25.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.374+0000 7fbd8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:25.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.374+0000 7fbd8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:25.352 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:25.352 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:25.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.377+0000 7fbdaa015640 10 librados: watch_flush enter 2024-03-29T11:14:25.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 10 librados: watch_flush exit 2024-03-29T11:14:25.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:14:25.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:14:25.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 >> v1:172.21.15.110:6816/1087970004 conn(0x7fbd9006dc90 legacy=0x7fbd90070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 10 monclient: shutdown 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 >> v1:172.21.15.105:6789/0 conn(0x7fbda41457f0 legacy=0x7fbda41d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 shutdown_connections 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 >> 172.21.15.105:0/2890500854 conn(0x7fbda4066a00 msgr2=0x7fbda40b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 shutdown_connections 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 -- 172.21.15.105:0/2890500854 wait complete. 2024-03-29T11:14:25.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:25.378+0000 7fbdaa015640 1 librados: shutdown 2024-03-29T11:14:25.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:32.181 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:32.325 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.347+0000 7f57d4974640 10 monclient: get_monmap_and_config 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.347+0000 7f57d4974640 10 monclient: build_initial_monmap 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.347+0000 7f57d4974640 10 monclient: monmap: 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:32.348739+0000 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:32.348739+0000 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:32.326 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:32.327 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:32.327 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.348+0000 7f57d4974640 1 Processor -- start 2024-03-29T11:14:32.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.348+0000 7f57d4974640 1 -- start start 2024-03-29T11:14:32.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.348+0000 7f57d4974640 10 monclient: init 2024-03-29T11:14:32.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:32.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-b con 0x7f57c8006e00 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-a con 0x7f57c8035dd0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80c9640 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8035940 con 0x7f57c8035dd0 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8036250 con 0x7f57c80c9640 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8035610 con 0x7f57c8006e00 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:32.330 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.349+0000 7f57d4974640 10 monclient(hunting): authenticate will time out at 1146.625863s 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57cf7fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f57c8006e00 0x7f57c80cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:57290/0 (socket says 172.21.15.105:57290) 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57cf7fe640 1 -- 172.21.15.105:0/479923404 learned_addr learned my addr 172.21.15.105:0/479923404 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493900431 0 0) 0x7f57c8035610 con 0x7f57c8006e00 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:14:32.331 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.352+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c0003520 con 0x7f57c8006e00 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117321782 0 0) 0x7f57c0003520 con 0x7f57c8006e00 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 >> v1:172.21.15.110:6789/0 conn(0x7f57c80c9640 legacy=0x7f57c80d7530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 >> v1:172.21.15.105:6789/0 conn(0x7f57c8035dd0 legacy=0x7f57c80c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c8034df0 con 0x7f57c8006e00 2024-03-29T11:14:32.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f57d004f730 con 0x7f57c8006e00 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57ce7fc640 10 monclient: dump: 2024-03-29T11:14:32.338 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57d4974640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.354+0000 7f57d4974640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:32.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d0075f10 con 0x7f57c8006e00 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 1 -- 172.21.15.105:0/479923404 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f57d00758b0 con 0x7f57c8006e00 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57ce7fc640 10 monclient: dump: 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.340 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57d4974640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.355+0000 7f57d4974640 4 set_mon_vals no callback set 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.356+0000 7f57d4974640 10 monclient: shutdown 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.356+0000 7f57d4974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.356+0000 7f57d4974640 1 -- 172.21.15.105:0/479923404 >> v1:172.21.15.105:6790/0 conn(0x7f57c8006e00 legacy=0x7f57c80cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.356+0000 7f57d4974640 1 -- 172.21.15.105:0/479923404 shutdown_connections 2024-03-29T11:14:32.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.356+0000 7f57d4974640 1 -- 172.21.15.105:0/479923404 >> 172.21.15.105:0/479923404 conn(0x7f57c8005bd0 msgr2=0x7f57c8005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.359+0000 7f57d4974640 1 -- 172.21.15.105:0/479923404 shutdown_connections 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.359+0000 7f57d4974640 1 -- 172.21.15.105:0/479923404 wait complete. 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 10 monclient: build_initial_monmap 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 10 monclient: monmap: 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:32.346 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 librados: starting msgr at 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 librados: starting objecter 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 Processor -- start 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 -- start start 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 librados: setting wanted keys 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 1 librados: calling monclient init 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.360+0000 7f57d4974640 10 monclient: init 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-a con 0x7f57c8006e00 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:32.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-b con 0x7f57c8035dd0 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80c9640 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80cffb0 con 0x7f57c8006e00 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c802eb40 con 0x7f57c8035dd0 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c802ecd0 con 0x7f57c80c9640 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.361+0000 7f57d4974640 10 monclient(hunting): authenticate will time out at 1146.637640s 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.364+0000 7f57ceffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f57c8035dd0 0x7f57c816d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:49399/0 (socket says 172.21.15.105:49399) 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.364+0000 7f57ceffd640 1 -- 172.21.15.105:0/41059551 learned_addr learned my addr 172.21.15.105:0/41059551 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.364+0000 7f57bbfff640 10 client.?.objecter ms_handle_connect 0x7f57c8035dd0 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.364+0000 7f57bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 client.?.objecter ms_handle_connect 0x7f57c80c9640 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825083830 0 0) 0x7f57c802eb40 con 0x7f57c8035dd0 2024-03-29T11:14:32.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4004eb0 con 0x7f57c8035dd0 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597566987 0 0) 0x7f57c802ecd0 con 0x7f57c80c9640 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.365+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4006230 con 0x7f57c80c9640 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626507032 0 0) 0x7f57b4006230 con 0x7f57c80c9640 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57c802ecd0 con 0x7f57c80c9640 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f57c4001710 con 0x7f57c80c9640 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient(hunting): dump: 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1904860841 0 0) 0x7f57c802ecd0 con 0x7f57c80c9640 2024-03-29T11:14:32.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 >> v1:172.21.15.110:6789/0 conn(0x7f57c8035dd0 legacy=0x7f57c816d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 >> v1:172.21.15.105:6789/0 conn(0x7f57c8006e00 legacy=0x7f57c816cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:32.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 monclient: found mon.c 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c802ee60 con 0x7f57c80c9640 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57d4974640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57d4974640 10 monclient: _renew_subs 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57d4974640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c802f110 con 0x7f57c80c9640 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57c4001d60 con 0x7f57c80c9640 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f57c4003790 con 0x7f57c80c9640 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:32.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57bbfff640 10 monclient: dump: 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:32.353 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.366+0000 7f57cdffb640 4 set_mon_vals no callback set 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.367+0000 7f57d4974640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.367+0000 7f57d4974640 10 monclient: _renew_subs 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.367+0000 7f57d4974640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.367+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c81751c0 con 0x7f57c80c9640 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.367+0000 7f57d4974640 1 librados: init done 2024-03-29T11:14:32.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.368+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f57c4019760 con 0x7f57c80c9640 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57d4974640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57d4974640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57d4974640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c8007b40 con 0x7f57c80c9640 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f57c4053400 con 0x7f57c80c9640 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57bbfff640 10 client.4502.objecter ms_dispatch 0x7f57d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:32.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57bbfff640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57bbfff640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57cf7fe640 10 monclient: get_auth_request con 0x7f57b406e160 auth_method 0 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.369+0000 7f57bbfff640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.370+0000 7f57bbfff640 10 client.4502.objecter ms_handle_connect 0x7f57b406e160 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.377+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57c404f6a0 con 0x7f57c80c9640 2024-03-29T11:14:32.356 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.377+0000 7f57bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:32.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.377+0000 7f57bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:32.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.698+0000 7f57d4974640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.698+0000 7f57d4974640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.698+0000 7f57d4974640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:14:32.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.698+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57c80cc320 con 0x7f57c80c9640 2024-03-29T11:14:32.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.699+0000 7f57bbfff640 1 -- 172.21.15.105:0/41059551 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f57c404f6a0 con 0x7f57c80c9640 2024-03-29T11:14:32.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.699+0000 7f57bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:32.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.699+0000 7f57bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:32.679 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:32.679 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.704+0000 7f57d4974640 10 librados: watch_flush enter 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.704+0000 7f57d4974640 10 librados: watch_flush exit 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.704+0000 7f57d4974640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.704+0000 7f57d4974640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.704+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 >> v1:172.21.15.110:6816/1087970004 conn(0x7f57b406e160 legacy=0x7f57b4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.705+0000 7f57d4974640 10 monclient: shutdown 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.705+0000 7f57d4974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:32.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.705+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 >> v1:172.21.15.105:6790/0 conn(0x7f57c80c9640 legacy=0x7f57c802e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:32.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.706+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 shutdown_connections 2024-03-29T11:14:32.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.706+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 >> 172.21.15.105:0/41059551 conn(0x7f57c8005bd0 msgr2=0x7f57c80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:32.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.707+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 shutdown_connections 2024-03-29T11:14:32.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.707+0000 7f57d4974640 1 -- 172.21.15.105:0/41059551 wait complete. 2024-03-29T11:14:32.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:32.707+0000 7f57d4974640 1 librados: shutdown 2024-03-29T11:14:32.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:39.509 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.680+0000 7fb3309e1640 10 monclient: get_monmap_and_config 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.680+0000 7fb3309e1640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.680+0000 7fb3309e1640 10 monclient: monmap: 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:39.658 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:39.681401+0000 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:39.681401+0000 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:39.659 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:39.660 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.662 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.684+0000 7fb3309e1640 1 Processor -- start 2024-03-29T11:14:39.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.685+0000 7fb3309e1640 1 -- start start 2024-03-29T11:14:39.668 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.685+0000 7fb3309e1640 10 monclient: init 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3240389e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-a con 0x7fb324038dc0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-b con 0x7fb324030660 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240c54c0 con 0x7fb324038dc0 2024-03-29T11:14:39.670 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb324036430 con 0x7fb3240389e0 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb324039840 con 0x7fb324030660 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.686+0000 7fb3309e1640 10 monclient(hunting): authenticate will time out at 1153.962619s 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32bfff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fb324030660 0x7fb3240d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:40537/0 (socket says 172.21.15.105:40537) 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32bfff640 1 -- 172.21.15.105:0/1105552575 learned_addr learned my addr 172.21.15.105:0/1105552575 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469704588 0 0) 0x7fb324039840 con 0x7fb324030660 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c003520 con 0x7fb324030660 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1347642587 0 0) 0x7fb31c003520 con 0x7fb324030660 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 >> v1:172.21.15.110:6789/0 conn(0x7fb3240389e0 legacy=0x7fb324033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:39.671 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 >> v1:172.21.15.105:6789/0 conn(0x7fb324038dc0 legacy=0x7fb3240cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb324035690 con 0x7fb324030660 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb318001690 con 0x7fb324030660 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:39.672 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: dump: 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:39.673 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb3309e1640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb3309e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb318001860 con 0x7fb324030660 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 1 -- 172.21.15.105:0/1105552575 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb318001cb0 con 0x7fb324030660 2024-03-29T11:14:39.674 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb32a7fc640 10 monclient: dump: 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb3309e1640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:39.675 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.691+0000 7fb3309e1640 4 set_mon_vals no callback set 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.692+0000 7fb3309e1640 10 monclient: shutdown 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.692+0000 7fb3309e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:39.676 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.692+0000 7fb3309e1640 1 -- 172.21.15.105:0/1105552575 >> v1:172.21.15.105:6790/0 conn(0x7fb324030660 legacy=0x7fb3240d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.693+0000 7fb3309e1640 1 -- 172.21.15.105:0/1105552575 shutdown_connections 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.693+0000 7fb3309e1640 1 -- 172.21.15.105:0/1105552575 >> 172.21.15.105:0/1105552575 conn(0x7fb324005c00 msgr2=0x7fb324006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.697+0000 7fb3309e1640 1 -- 172.21.15.105:0/1105552575 shutdown_connections 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.697+0000 7fb3309e1640 1 -- 172.21.15.105:0/1105552575 wait complete. 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.698+0000 7fb3309e1640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.698+0000 7fb3309e1640 10 monclient: monmap: 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.678 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.698+0000 7fb3309e1640 1 librados: starting msgr at 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.698+0000 7fb3309e1640 1 librados: starting objecter 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.698+0000 7fb3309e1640 1 Processor -- start 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.699+0000 7fb3309e1640 1 -- start start 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.699+0000 7fb3309e1640 1 librados: setting wanted keys 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.699+0000 7fb3309e1640 1 librados: calling monclient init 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.699+0000 7fb3309e1640 10 monclient: init 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.700+0000 7fb3309e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.700+0000 7fb3309e1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.704+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-b con 0x7fb324030660 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:39.681 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.704+0000 7fb32b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb324030660 0x7fb3240b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:39477/0 (socket says 172.21.15.105:39477) 2024-03-29T11:14:39.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.704+0000 7fb32b7fe640 1 -- 172.21.15.105:0/2416376510 learned_addr learned my addr 172.21.15.105:0/2416376510 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3240389e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3240c5650 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240c5b70 con 0x7fb3240389e0 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240caa70 con 0x7fb324030660 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240b19b0 con 0x7fb3240c5650 2024-03-29T11:14:39.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb3309e1640 10 monclient(hunting): authenticate will time out at 1153.981854s 2024-03-29T11:14:39.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb317fff640 10 client.?.objecter ms_handle_connect 0x7fb324030660 2024-03-29T11:14:39.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.705+0000 7fb317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625626847 0 0) 0x7fb3240caa70 con 0x7fb324030660 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb310004eb0 con 0x7fb324030660 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 client.?.objecter ms_handle_connect 0x7fb3240389e0 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.710+0000 7fb317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373085255 0 0) 0x7fb3240c5b70 con 0x7fb3240389e0 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3240caa70 con 0x7fb3240389e0 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762770479 0 0) 0x7fb3240caa70 con 0x7fb3240389e0 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3240c5b70 con 0x7fb3240389e0 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb32c04fdd0 con 0x7fb3240389e0 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): dump: 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:39.691 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:39.692 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:39.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 client.?.objecter ms_handle_connect 0x7fb3240c5650 2024-03-29T11:14:39.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2257580920 0 0) 0x7fb3240c5b70 con 0x7fb3240389e0 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 >> v1:172.21.15.105:6790/0 conn(0x7fb3240c5650 legacy=0x7fb3240c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 >> v1:172.21.15.110:6789/0 conn(0x7fb324030660 legacy=0x7fb3240b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 monclient: found mon.a 2024-03-29T11:14:39.695 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3240b1b40 con 0x7fb3240389e0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.711+0000 7fb317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 10 monclient: _renew_subs 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3240ca5c0 con 0x7fb3240389e0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 10 monclient: _renew_subs 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3240ca8c0 con 0x7fb3240389e0 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb3309e1640 1 librados: init done 2024-03-29T11:14:39.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb32c04f400 con 0x7fb3240389e0 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb32c075f10 con 0x7fb3240389e0 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.712+0000 7fb317fff640 10 monclient: dump: 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:39.697 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:39.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.713+0000 7fb3297fa640 4 set_mon_vals no callback set 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.714+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb32c067bf0 con 0x7fb3240389e0 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.714+0000 7fb3309e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fb32c0fe3b0 con 0x7fb3240389e0 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb317fff640 10 client.4440.objecter ms_dispatch 0x7fb32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb317fff640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb317fff640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb317fff640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb3309e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb3309e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c005080 con 0x7fb3240389e0 2024-03-29T11:14:39.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.716+0000 7fb32b7fe640 10 monclient: get_auth_request con 0x7fb31006db70 auth_method 0 2024-03-29T11:14:39.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.720+0000 7fb317fff640 10 client.4440.objecter ms_handle_connect 0x7fb31006db70 2024-03-29T11:14:39.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.728+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb32c0c9c90 con 0x7fb3240389e0 2024-03-29T11:14:39.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.728+0000 7fb317fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:39.728+0000 7fb317fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.054+0000 7fb3309e1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.054+0000 7fb3309e1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.054+0000 7fb3309e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.054+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb30c005330 con 0x7fb3240389e0 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.056+0000 7fb317fff640 1 -- 172.21.15.105:0/2416376510 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fb32c0cbbc0 con 0x7fb3240389e0 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.056+0000 7fb317fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.056+0000 7fb317fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:40.034 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:40.035 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 10 librados: watch_flush enter 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 10 librados: watch_flush exit 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb31006db70 legacy=0x7fb310070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 10 monclient: shutdown 2024-03-29T11:14:40.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:40.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.059+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 >> v1:172.21.15.105:6789/0 conn(0x7fb3240389e0 legacy=0x7fb3240c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.060+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 shutdown_connections 2024-03-29T11:14:40.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.060+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 >> 172.21.15.105:0/2416376510 conn(0x7fb324005c00 msgr2=0x7fb32403ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:40.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.060+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 shutdown_connections 2024-03-29T11:14:40.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.060+0000 7fb3309e1640 1 -- 172.21.15.105:0/2416376510 wait complete. 2024-03-29T11:14:40.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:40.060+0000 7fb3309e1640 1 librados: shutdown 2024-03-29T11:14:40.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:46.862 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.036+0000 7fa92c8ff640 10 monclient: get_monmap_and_config 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.036+0000 7fa92c8ff640 10 monclient: build_initial_monmap 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.036+0000 7fa92c8ff640 10 monclient: monmap: 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:47.037939+0000 2024-03-29T11:14:47.015 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:47.037939+0000 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:47.016 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:47.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.038+0000 7fa92c8ff640 1 Processor -- start 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.038+0000 7fa92c8ff640 1 -- start start 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.038+0000 7fa92c8ff640 10 monclient: init 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.039+0000 7fa92c8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.039+0000 7fa92c8ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.039+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-c con 0x7fa9280ae250 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.039+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-b con 0x7fa92813f970 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-a con 0x7fa928142800 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa926d9d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fa928142800 0x7fa92814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:59806/0 (socket says 172.21.15.105:59806) 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa926d9d640 1 -- 172.21.15.105:0/2000424526 learned_addr learned my addr 172.21.15.105:0/2000424526 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280afe00 con 0x7fa928142800 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928142c10 con 0x7fa9280ae250 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280afb20 con 0x7fa92813f970 2024-03-29T11:14:47.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92c8ff640 10 monclient(hunting): authenticate will time out at 1161.316235s 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928922727 0 0) 0x7fa9280afb20 con 0x7fa92813f970 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa910003520 con 0x7fa92813f970 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273660381 0 0) 0x7fa9280afe00 con 0x7fa928142800 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9280afb20 con 0x7fa928142800 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.040+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794620425 0 0) 0x7fa9280afb20 con 0x7fa928142800 2024-03-29T11:14:47.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 >> v1:172.21.15.105:6790/0 conn(0x7fa92813f970 legacy=0x7fa9281420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 >> v1:172.21.15.110:6789/0 conn(0x7fa9280ae250 legacy=0x7fa92813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 monclient: found mon.noname-a 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280aab80 con 0x7fa928142800 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fa9200019f0 con 0x7fa928142800 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: dump: 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:47.024 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92c8ff640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92c8ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa920001c40 con 0x7fa928142800 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92559a640 1 -- 172.21.15.105:0/2000424526 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fa920001fd0 con 0x7fa928142800 2024-03-29T11:14:47.025 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92c8ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.041+0000 7fa92c8ff640 4 set_mon_vals no callback set 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 10 monclient: shutdown 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 >> v1:172.21.15.105:6789/0 conn(0x7fa928142800 legacy=0x7fa92814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 shutdown_connections 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 >> 172.21.15.105:0/2000424526 conn(0x7fa9280b35e0 msgr2=0x7fa9280b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 shutdown_connections 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.042+0000 7fa92c8ff640 1 -- 172.21.15.105:0/2000424526 wait complete. 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.043+0000 7fa92c8ff640 10 monclient: build_initial_monmap 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.043+0000 7fa92c8ff640 10 monclient: monmap: 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:47.026 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.043+0000 7fa92c8ff640 1 librados: starting msgr at 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.043+0000 7fa92c8ff640 1 librados: starting objecter 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 1 Processor -- start 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 1 -- start start 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 1 librados: setting wanted keys 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 1 librados: calling monclient init 2024-03-29T11:14:47.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 10 monclient: init 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.044+0000 7fa92c8ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-c con 0x7fa9280ae250 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-b con 0x7fa92813f970 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:47.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92659c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fa9280ae250 0x7fa928144700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:34464/0 (socket says 172.21.15.105:34464) 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92659c640 1 -- 172.21.15.105:0/4188841826 learned_addr learned my addr 172.21.15.105:0/4188841826 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa916ffd640 10 client.?.objecter ms_handle_connect 0x7fa9280ae250 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa916ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa916ffd640 10 client.?.objecter ms_handle_connect 0x7fa92813f970 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa916ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 10 monclient(hunting): picked mon.noname-a con 0x7fa928142800 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928145de0 con 0x7fa928142800 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928144e10 con 0x7fa92813f970 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92813fef0 con 0x7fa9280ae250 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.045+0000 7fa92c8ff640 10 monclient(hunting): authenticate will time out at 1161.321745s 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 client.?.objecter ms_handle_connect 0x7fa928142800 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106592029 0 0) 0x7fa92813fef0 con 0x7fa9280ae250 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa908004eb0 con 0x7fa9280ae250 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120290926 0 0) 0x7fa928144e10 con 0x7fa92813f970 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92813fef0 con 0x7fa92813f970 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348567745 0 0) 0x7fa92813fef0 con 0x7fa92813f970 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa928144e10 con 0x7fa92813f970 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fa91c001740 con 0x7fa92813f970 2024-03-29T11:14:47.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751766314 0 0) 0x7fa928145de0 con 0x7fa928142800 2024-03-29T11:14:47.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.046+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92813fef0 con 0x7fa928142800 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 926180028 0 0) 0x7fa928144e10 con 0x7fa92813f970 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 >> v1:172.21.15.105:6790/0 conn(0x7fa9280ae250 legacy=0x7fa928144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 >> v1:172.21.15.105:6789/0 conn(0x7fa928142800 legacy=0x7fa9281e1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 1 monclient: found mon.b 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928140080 con 0x7fa92813f970 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa916ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 10 monclient: _renew_subs 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9281e4f90 con 0x7fa92813f970 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 10 monclient: _renew_subs 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa928143c40 con 0x7fa92813f970 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.047+0000 7fa92c8ff640 1 librados: init done 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa91c001c10 con 0x7fa92813f970 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fa91c003610 con 0x7fa92813f970 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:47.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa916ffd640 10 monclient: dump: 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.048+0000 7fa924d99640 4 set_mon_vals no callback set 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.049+0000 7fa92c8ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.049+0000 7fa92c8ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.049+0000 7fa92c8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:47.038 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.049+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c005080 con 0x7fa92813f970 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.050+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fa91c0195e0 con 0x7fa92813f970 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.051+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fa91c0834f0 con 0x7fa92813f970 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.051+0000 7fa916ffd640 10 client.4447.objecter ms_dispatch 0x7fa928000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.051+0000 7fa916ffd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.051+0000 7fa916ffd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.051+0000 7fa916ffd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.052+0000 7fa92659c640 10 monclient: get_auth_request con 0x7fa90806dc90 auth_method 0 2024-03-29T11:14:47.039 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.052+0000 7fa916ffd640 10 client.4447.objecter ms_handle_connect 0x7fa90806dc90 2024-03-29T11:14:47.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.059+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa91c04ec00 con 0x7fa92813f970 2024-03-29T11:14:47.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.059+0000 7fa916ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.059+0000 7fa916ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:47.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.419+0000 7fa92c8ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.419+0000 7fa92c8ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.419+0000 7fa92c8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:47.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.419+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa90c005330 con 0x7fa92813f970 2024-03-29T11:14:47.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.420+0000 7fa916ffd640 1 -- 172.21.15.105:0/4188841826 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fa91c050d00 con 0x7fa92813f970 2024-03-29T11:14:47.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.420+0000 7fa916ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:47.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.420+0000 7fa916ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:47.398 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:47.398 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:47.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.423+0000 7fa92c8ff640 10 librados: watch_flush enter 2024-03-29T11:14:47.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.423+0000 7fa92c8ff640 10 librados: watch_flush exit 2024-03-29T11:14:47.401 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.423+0000 7fa92c8ff640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.423+0000 7fa92c8ff640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 >> v1:172.21.15.110:6816/1087970004 conn(0x7fa90806dc90 legacy=0x7fa908070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 10 monclient: shutdown 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 >> v1:172.21.15.110:6789/0 conn(0x7fa92813f970 legacy=0x7fa9281e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 shutdown_connections 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 >> 172.21.15.105:0/4188841826 conn(0x7fa9280b35e0 msgr2=0x7fa9280b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 shutdown_connections 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 -- 172.21.15.105:0/4188841826 wait complete. 2024-03-29T11:14:47.402 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:47.424+0000 7fa92c8ff640 1 librados: shutdown 2024-03-29T11:14:47.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:54.225 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.418+0000 7ffb77782640 10 monclient: get_monmap_and_config 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.418+0000 7ffb77782640 10 monclient: build_initial_monmap 2024-03-29T11:14:54.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.418+0000 7ffb77782640 10 monclient: monmap: 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:14:54.419860+0000 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:14:54.419860+0000 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:14:54.397 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.398 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.419+0000 7ffb77782640 1 Processor -- start 2024-03-29T11:14:54.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.423+0000 7ffb77782640 1 -- start start 2024-03-29T11:14:54.400 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.423+0000 7ffb77782640 10 monclient: init 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-b con 0x7ffb701422b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-a con 0x7ffb70142690 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-c con 0x7ffb700b17d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb70142c80 con 0x7ffb70142690 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aae40 con 0x7ffb700b17d0 2024-03-29T11:14:54.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7013fbd0 con 0x7ffb701422b0 2024-03-29T11:14:54.407 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:54.407 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.424+0000 7ffb77782640 10 monclient(hunting): authenticate will time out at 1168.700124s 2024-03-29T11:14:54.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.430+0000 7ffb75d1f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffb700b17d0 0x7ffb700b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:50283/0 (socket says 172.21.15.105:50283) 2024-03-29T11:14:54.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.430+0000 7ffb75d1f640 1 -- 172.21.15.105:0/1687503484 learned_addr learned my addr 172.21.15.105:0/1687503484 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:54.410 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.428+0000 7ffb74d1d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ffb70142690 0x7ffb700b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:52813/0 (socket says 172.21.15.105:52813) 2024-03-29T11:14:54.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455246269 0 0) 0x7ffb700aae40 con 0x7ffb700b17d0 2024-03-29T11:14:54.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.413 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb5c003520 con 0x7ffb700b17d0 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314615356 0 0) 0x7ffb70142c80 con 0x7ffb70142690 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:14:54.414 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.434+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700aae40 con 0x7ffb70142690 2024-03-29T11:14:54.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.437+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348431372 0 0) 0x7ffb5c003520 con 0x7ffb700b17d0 2024-03-29T11:14:54.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.437+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 >> v1:172.21.15.105:6790/0 conn(0x7ffb701422b0 legacy=0x7ffb7013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.437+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 >> v1:172.21.15.105:6789/0 conn(0x7ffb70142690 legacy=0x7ffb700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.417 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.437+0000 7ffb67fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.437+0000 7ffb67fff640 1 monclient: found mon.noname-c 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70151940 con 0x7ffb700b17d0 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb60001690 con 0x7ffb700b17d0 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb67fff640 10 monclient: dump: 2024-03-29T11:14:54.418 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb77782640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:14:54.419 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.438+0000 7ffb77782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb600017f0 con 0x7ffb700b17d0 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 1 -- 172.21.15.105:0/1687503484 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb60001bc0 con 0x7ffb700b17d0 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb67fff640 10 monclient: dump: 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.420 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.421 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.421 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.421 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb77782640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.439+0000 7ffb77782640 4 set_mon_vals no callback set 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.440+0000 7ffb77782640 10 monclient: shutdown 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.440+0000 7ffb77782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.440+0000 7ffb77782640 1 -- 172.21.15.105:0/1687503484 >> v1:172.21.15.110:6789/0 conn(0x7ffb700b17d0 legacy=0x7ffb700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.441+0000 7ffb77782640 1 -- 172.21.15.105:0/1687503484 shutdown_connections 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.441+0000 7ffb77782640 1 -- 172.21.15.105:0/1687503484 >> 172.21.15.105:0/1687503484 conn(0x7ffb70066a00 msgr2=0x7ffb70066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:54.423 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.442+0000 7ffb77782640 1 -- 172.21.15.105:0/1687503484 shutdown_connections 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.442+0000 7ffb77782640 1 -- 172.21.15.105:0/1687503484 wait complete. 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 10 monclient: build_initial_monmap 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 10 monclient: monmap: 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.424 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 1 librados: starting msgr at 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 1 librados: starting objecter 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 1 Processor -- start 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.443+0000 7ffb77782640 1 -- start start 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 1 librados: setting wanted keys 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 1 librados: calling monclient init 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 10 monclient: init 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:54.425 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:54.426 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.444+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-a con 0x7ffb700b17d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-c con 0x7ffb701422b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 10 monclient(hunting): picked mon.noname-b con 0x7ffb70142690 addr v1:172.21.15.110:6789/0 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700b6220 con 0x7ffb700b17d0 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700b6500 con 0x7ffb70142690 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7014d270 con 0x7ffb701422b0 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.448+0000 7ffb77782640 10 monclient(hunting): authenticate will time out at 1168.724724s 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.449+0000 7ffb7551e640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ffb700b17d0 0x7ffb700b6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:53373/0 (socket says 172.21.15.105:53373) 2024-03-29T11:14:54.428 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.449+0000 7ffb7551e640 1 -- 172.21.15.105:0/331972182 learned_addr learned my addr 172.21.15.105:0/331972182 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:14:54.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.451+0000 7ffb65ffb640 10 client.?.objecter ms_handle_connect 0x7ffb700b17d0 2024-03-29T11:14:54.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.451+0000 7ffb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.452+0000 7ffb65ffb640 10 client.?.objecter ms_handle_connect 0x7ffb701422b0 2024-03-29T11:14:54.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.452+0000 7ffb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 client.?.objecter ms_handle_connect 0x7ffb70142690 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087887785 0 0) 0x7ffb7014d270 con 0x7ffb701422b0 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb701422b0 2024-03-29T11:14:54.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611474363 0 0) 0x7ffb700b6500 con 0x7ffb70142690 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb7014d270 con 0x7ffb70142690 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117242685 0 0) 0x7ffb700b6220 con 0x7ffb700b17d0 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:14:54.437 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.458+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700b6500 con 0x7ffb700b17d0 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108080493 0 0) 0x7ffb54004eb0 con 0x7ffb701422b0 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb700b6220 con 0x7ffb701422b0 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb6c001840 con 0x7ffb701422b0 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.438 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4195786889 0 0) 0x7ffb7014d270 con 0x7ffb70142690 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb70142690 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb60001b00 con 0x7ffb70142690 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.439 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732747120 0 0) 0x7ffb700b6500 con 0x7ffb700b17d0 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb7014d270 con 0x7ffb700b17d0 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb68001c30 con 0x7ffb700b17d0 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.459+0000 7ffb65ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.440 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1325527029 0 0) 0x7ffb54004eb0 con 0x7ffb70142690 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 >> v1:172.21.15.105:6790/0 conn(0x7ffb701422b0 legacy=0x7ffb7014dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 >> v1:172.21.15.105:6789/0 conn(0x7ffb700b17d0 legacy=0x7ffb700b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 1 monclient: found mon.b 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:54.441 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7014d400 con 0x7ffb70142690 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 10 monclient: _renew_subs 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb701310b0 con 0x7ffb70142690 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 10 monclient: _renew_subs 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb701315e0 con 0x7ffb70142690 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.460+0000 7ffb77782640 1 librados: init done 2024-03-29T11:14:54.442 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60001c60 con 0x7ffb70142690 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ffb60004a90 con 0x7ffb70142690 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb65ffb640 10 monclient: dump: 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:14:54.443 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.461+0000 7ffb677fe640 4 set_mon_vals no callback set 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.463+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ffb6001aa60 con 0x7ffb70142690 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ffb60084920 con 0x7ffb70142690 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb65ffb640 10 client.4453.objecter ms_dispatch 0x7ffb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb65ffb640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb65ffb640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb65ffb640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.464+0000 7ffb77782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.465+0000 7ffb7551e640 10 monclient: get_auth_request con 0x7ffb5406e300 auth_method 0 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.465+0000 7ffb77782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.465+0000 7ffb77782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.444 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.465+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb50005080 con 0x7ffb70142690 2024-03-29T11:14:54.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.465+0000 7ffb65ffb640 10 client.4453.objecter ms_handle_connect 0x7ffb5406e300 2024-03-29T11:14:54.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.474+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb60050200 con 0x7ffb70142690 2024-03-29T11:14:54.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.474+0000 7ffb65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:54.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.474+0000 7ffb65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:54.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.821+0000 7ffb77782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.821+0000 7ffb77782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.822+0000 7ffb77782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:14:54.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.822+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb50005330 con 0x7ffb70142690 2024-03-29T11:14:54.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.823+0000 7ffb65ffb640 1 -- 172.21.15.105:0/331972182 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ffb60052130 con 0x7ffb70142690 2024-03-29T11:14:54.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.823+0000 7ffb65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.823+0000 7ffb65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:54.802 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:14:54.803 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:54.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.828+0000 7ffb77782640 10 librados: watch_flush enter 2024-03-29T11:14:54.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.828+0000 7ffb77782640 10 librados: watch_flush exit 2024-03-29T11:14:54.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.828+0000 7ffb77782640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:14:54.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.828+0000 7ffb77782640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:14:54.809 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 >> v1:172.21.15.110:6816/1087970004 conn(0x7ffb5406e300 legacy=0x7ffb540707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 10 monclient: shutdown 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 >> v1:172.21.15.110:6789/0 conn(0x7ffb70142690 legacy=0x7ffb701e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 shutdown_connections 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 >> 172.21.15.105:0/331972182 conn(0x7ffb70066a00 msgr2=0x7ffb7013b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 shutdown_connections 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.831+0000 7ffb77782640 1 -- 172.21.15.105:0/331972182 wait complete. 2024-03-29T11:14:54.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:14:54.832+0000 7ffb77782640 1 librados: shutdown 2024-03-29T11:14:54.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:01.633 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:01.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.822+0000 7f5a7697c640 10 monclient: get_monmap_and_config 2024-03-29T11:15:01.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.822+0000 7f5a7697c640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.822+0000 7f5a7697c640 10 monclient: monmap: 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:01.823398+0000 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:01.823398+0000 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:01.801 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:01.802 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.805 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 1 Processor -- start 2024-03-29T11:15:01.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 1 -- start start 2024-03-29T11:15:01.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient: init 2024-03-29T11:15:01.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:01.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-c con 0x7f5a700b0860 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7013ef30 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.825+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700af8e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.826+0000 7f5a7697c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700b0c40 con 0x7f5a7013ef30 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.826+0000 7f5a7697c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700aed10 con 0x7f5a700b0860 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.826+0000 7f5a7697c640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700afd50 con 0x7f5a700af8e0 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.826+0000 7f5a7697c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.826+0000 7f5a7697c640 10 monclient(hunting): authenticate will time out at 1176.102004s 2024-03-29T11:15:01.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.828+0000 7f5a74f19640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f5a700af8e0 0x7f5a70149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41772/0 (socket says 172.21.15.105:41772) 2024-03-29T11:15:01.808 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.828+0000 7f5a74f19640 1 -- 172.21.15.105:0/970743106 learned_addr learned my addr 172.21.15.105:0/970743106 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:01.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.832+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398805874 0 0) 0x7f5a700aed10 con 0x7f5a700b0860 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003520 con 0x7f5a700b0860 2024-03-29T11:15:01.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632284463 0 0) 0x7f5a58003520 con 0x7f5a700b0860 2024-03-29T11:15:01.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 >> v1:172.21.15.105:6790/0 conn(0x7f5a700af8e0 legacy=0x7f5a70149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 >> v1:172.21.15.105:6789/0 conn(0x7f5a7013ef30 legacy=0x7f5a700af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:01.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 monclient: found mon.noname-c 2024-03-29T11:15:01.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:15:01.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700ab1b0 con 0x7f5a700b0860 2024-03-29T11:15:01.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a60001690 con 0x7f5a700b0860 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a6effd640 10 monclient: dump: 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.815 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a7697c640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.833+0000 7f5a7697c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.834+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a60001860 con 0x7f5a700b0860 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.834+0000 7f5a6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.834+0000 7f5a7697c640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.834+0000 7f5a7697c640 4 set_mon_vals no callback set 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a6effd640 1 -- 172.21.15.105:0/970743106 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a60001cb0 con 0x7f5a700b0860 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a6effd640 10 monclient: dump: 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.816 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.817 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.817 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.817 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.817 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.817 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a7697c640 10 monclient: shutdown 2024-03-29T11:15:01.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.835+0000 7f5a7697c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.836+0000 7f5a7697c640 1 -- 172.21.15.105:0/970743106 >> v1:172.21.15.110:6789/0 conn(0x7f5a700b0860 legacy=0x7f5a7013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.836+0000 7f5a7697c640 1 -- 172.21.15.105:0/970743106 shutdown_connections 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.836+0000 7f5a7697c640 1 -- 172.21.15.105:0/970743106 >> 172.21.15.105:0/970743106 conn(0x7f5a7006b660 msgr2=0x7f5a7006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.836+0000 7f5a7697c640 1 -- 172.21.15.105:0/970743106 shutdown_connections 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.837+0000 7f5a7697c640 1 -- 172.21.15.105:0/970743106 wait complete. 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.838+0000 7f5a7697c640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.838+0000 7f5a7697c640 10 monclient: monmap: 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.820 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.838+0000 7f5a7697c640 1 librados: starting msgr at 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.838+0000 7f5a7697c640 1 librados: starting objecter 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.838+0000 7f5a7697c640 1 Processor -- start 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.839+0000 7f5a7697c640 1 -- start start 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.839+0000 7f5a7697c640 1 librados: setting wanted keys 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.839+0000 7f5a7697c640 1 librados: calling monclient init 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.839+0000 7f5a7697c640 10 monclient: init 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-a con 0x7f5a700af8e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700b0860 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:01.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7013ef30 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7006f110 con 0x7f5a700af8e0 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7006ffe0 con 0x7f5a700b0860 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700702c0 con 0x7f5a7013ef30 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a74f19640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f5a7013ef30 0x7f5a701e6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:42674/0 (socket says 172.21.15.105:42674) 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a74f19640 1 -- 172.21.15.105:0/1191570185 learned_addr learned my addr 172.21.15.105:0/1191570185 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a7697c640 10 monclient(hunting): authenticate will time out at 1176.116591s 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a6cff9640 10 client.?.objecter ms_handle_connect 0x7f5a7013ef30 2024-03-29T11:15:01.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.840+0000 7f5a6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.843+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737343383 0 0) 0x7f5a700702c0 con 0x7f5a7013ef30 2024-03-29T11:15:01.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c004eb0 con 0x7f5a7013ef30 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 client.?.objecter ms_handle_connect 0x7f5a700af8e0 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.844+0000 7f5a6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708977654 0 0) 0x7f5a5c004eb0 con 0x7f5a7013ef30 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a5c005300 con 0x7f5a7013ef30 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a68001b10 con 0x7f5a7013ef30 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:01.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): dump: 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.827 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542379039 0 0) 0x7f5a7006f110 con 0x7f5a700af8e0 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.845+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c005850 con 0x7f5a700af8e0 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 client.?.objecter ms_handle_connect 0x7f5a700b0860 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444501061 0 0) 0x7f5a7006ffe0 con 0x7f5a700b0860 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7006f110 con 0x7f5a700b0860 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 883128184 0 0) 0x7f5a5c005850 con 0x7f5a700af8e0 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a7006ffe0 con 0x7f5a700af8e0 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a60001cb0 con 0x7f5a700af8e0 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): dump: 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.829 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 222562404 0 0) 0x7f5a7006f110 con 0x7f5a700b0860 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a5c005850 con 0x7f5a700b0860 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a640016f0 con 0x7f5a700b0860 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.846+0000 7f5a6cff9640 10 monclient(hunting): dump: 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.830 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1303105192 0 0) 0x7f5a5c005850 con 0x7f5a700b0860 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 >> v1:172.21.15.105:6790/0 conn(0x7f5a7013ef30 legacy=0x7f5a701e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 >> v1:172.21.15.105:6789/0 conn(0x7f5a700af8e0 legacy=0x7f5a7012f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 monclient: found mon.b 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700705a0 con 0x7f5a700b0860 2024-03-29T11:15:01.831 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 10 monclient: _renew_subs 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:01.832 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7006fad0 con 0x7f5a700b0860 2024-03-29T11:15:01.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 10 monclient: _renew_subs 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70130040 con 0x7f5a700b0860 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a7697c640 1 librados: init done 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a64001be0 con 0x7f5a700b0860 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5a64003790 con 0x7f5a700b0860 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6cff9640 10 monclient: dump: 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:01.834 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:01.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.847+0000 7f5a6dffb640 4 set_mon_vals no callback set 2024-03-29T11:15:01.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.849+0000 7f5a7697c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.849+0000 7f5a7697c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.849+0000 7f5a7697c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:01.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.849+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a50005080 con 0x7f5a700b0860 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.849+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f5a64019760 con 0x7f5a700b0860 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.850+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f5a640532a0 con 0x7f5a700b0860 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.850+0000 7f5a6cff9640 10 client.4459.objecter ms_dispatch 0x7f5a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.850+0000 7f5a6cff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.850+0000 7f5a6cff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.850+0000 7f5a6cff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.851+0000 7f5a6ffff640 10 monclient: get_auth_request con 0x7f5a5c06e7f0 auth_method 0 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.851+0000 7f5a6cff9640 10 client.4459.objecter ms_handle_connect 0x7f5a5c06e7f0 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.857+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a6404f540 con 0x7f5a700b0860 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.857+0000 7f5a6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:01.857+0000 7f5a6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:02.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.221+0000 7f5a7697c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.221+0000 7f5a7697c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.221+0000 7f5a7697c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:02.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.221+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a50005330 con 0x7f5a700b0860 2024-03-29T11:15:02.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.222+0000 7f5a6cff9640 1 -- 172.21.15.105:0/1191570185 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f5a6404f540 con 0x7f5a700b0860 2024-03-29T11:15:02.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.222+0000 7f5a6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:02.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.222+0000 7f5a6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:02.201 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:02.201 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:02.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.226+0000 7f5a7697c640 10 librados: watch_flush enter 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.226+0000 7f5a7697c640 10 librados: watch_flush exit 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.226+0000 7f5a7697c640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.226+0000 7f5a7697c640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 >> v1:172.21.15.110:6816/1087970004 conn(0x7f5a5c06e7f0 legacy=0x7f5a5c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 10 monclient: shutdown 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:02.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 >> v1:172.21.15.110:6789/0 conn(0x7f5a700b0860 legacy=0x7f5a701dea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 shutdown_connections 2024-03-29T11:15:02.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.227+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 >> 172.21.15.105:0/1191570185 conn(0x7f5a7006b660 msgr2=0x7f5a7013e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:02.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.229+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 shutdown_connections 2024-03-29T11:15:02.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.229+0000 7f5a7697c640 1 -- 172.21.15.105:0/1191570185 wait complete. 2024-03-29T11:15:02.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:02.229+0000 7f5a7697c640 1 librados: shutdown 2024-03-29T11:15:02.226 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:09.034 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.206+0000 7ff13cf3a640 10 monclient: get_monmap_and_config 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.206+0000 7ff13cf3a640 10 monclient: build_initial_monmap 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.206+0000 7ff13cf3a640 10 monclient: monmap: 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:09.207315+0000 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:09.207315+0000 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:09.184 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:09.185 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:09.185 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.207+0000 7ff13cf3a640 1 Processor -- start 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.207+0000 7ff13cf3a640 1 -- start start 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.207+0000 7ff13cf3a640 10 monclient: init 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:09.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1380b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff13814a700 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1380ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380aba70 con 0x7ff1380b0770 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380aecb0 con 0x7ff13814a700 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380af150 con 0x7ff1380ac5a0 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13cf3a640 10 monclient(hunting): authenticate will time out at 1183.484658s 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13659c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff1380b0770 0x7ff138149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:50916/0 (socket says 172.21.15.105:50916) 2024-03-29T11:15:09.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.208+0000 7ff13659c640 1 -- 172.21.15.105:0/2916205587 learned_addr learned my addr 172.21.15.105:0/2916205587 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:09.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338712720 0 0) 0x7ff1380af150 con 0x7ff1380ac5a0 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124003520 con 0x7ff1380ac5a0 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462464245 0 0) 0x7ff1380aba70 con 0x7ff1380b0770 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.209+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1380af150 con 0x7ff1380b0770 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938856953 0 0) 0x7ff1380af150 con 0x7ff1380b0770 2024-03-29T11:15:09.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 >> v1:172.21.15.105:6790/0 conn(0x7ff1380ac5a0 legacy=0x7ff138151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 >> v1:172.21.15.110:6789/0 conn(0x7ff13814a700 legacy=0x7ff1380abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 monclient: found mon.noname-a 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380ac9d0 con 0x7ff1380b0770 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: _finish_auth 0 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff128001670 con 0x7ff1380b0770 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 10 monclient: dump: 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.192 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff13cf3a640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.210+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1280018c0 con 0x7ff1380b0770 2024-03-29T11:15:09.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff13cf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff135d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff135d9b640 1 -- 172.21.15.105:0/2916205587 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff128001c50 con 0x7ff1380b0770 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff13cf3a640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff13cf3a640 4 set_mon_vals no callback set 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff135d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff135d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.211+0000 7ff135d9b640 10 monclient: dump: 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.194 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 10 monclient: shutdown 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 1 -- 172.21.15.105:0/2916205587 >> v1:172.21.15.105:6789/0 conn(0x7ff1380b0770 legacy=0x7ff138149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 1 -- 172.21.15.105:0/2916205587 shutdown_connections 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 1 -- 172.21.15.105:0/2916205587 >> 172.21.15.105:0/2916205587 conn(0x7ff13806b660 msgr2=0x7ff13806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 1 -- 172.21.15.105:0/2916205587 shutdown_connections 2024-03-29T11:15:09.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.212+0000 7ff13cf3a640 1 -- 172.21.15.105:0/2916205587 wait complete. 2024-03-29T11:15:09.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.214+0000 7ff13cf3a640 10 monclient: build_initial_monmap 2024-03-29T11:15:09.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.214+0000 7ff13cf3a640 10 monclient: monmap: 2024-03-29T11:15:09.200 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.214+0000 7ff13cf3a640 1 librados: starting msgr at 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.214+0000 7ff13cf3a640 1 librados: starting objecter 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.215+0000 7ff13cf3a640 1 Processor -- start 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.215+0000 7ff13cf3a640 1 -- start start 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.215+0000 7ff13cf3a640 1 librados: setting wanted keys 2024-03-29T11:15:09.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.215+0000 7ff13cf3a640 1 librados: calling monclient init 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.215+0000 7ff13cf3a640 10 monclient: init 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1380ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1380b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff12dd9b640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff1380b0770 0x7ff1381db8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:51328/0 (socket says 172.21.15.105:51328) 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff13814a700 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff12dd9b640 1 -- 172.21.15.105:0/106913354 learned_addr learned my addr 172.21.15.105:0/106913354 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.216+0000 7ff13cf3a640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff138127440 con 0x7ff1380b0770 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff13cf3a640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff138127f90 con 0x7ff1380ac5a0 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff13cf3a640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff138128270 con 0x7ff13814a700 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff13cf3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff13cf3a640 10 monclient(hunting): authenticate will time out at 1183.493245s 2024-03-29T11:15:09.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter ms_handle_connect 0x7ff1380b0770 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter ms_handle_connect 0x7ff13814a700 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235611744 0 0) 0x7ff138127440 con 0x7ff1380b0770 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff118004eb0 con 0x7ff1380b0770 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter ms_handle_connect 0x7ff1380ac5a0 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.217+0000 7ff12f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275079914 0 0) 0x7ff138128270 con 0x7ff13814a700 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138127440 con 0x7ff13814a700 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463659651 0 0) 0x7ff138127f90 con 0x7ff1380ac5a0 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138128270 con 0x7ff1380ac5a0 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842017660 0 0) 0x7ff118004eb0 con 0x7ff1380b0770 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff138127f90 con 0x7ff1380b0770 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff120001910 con 0x7ff1380b0770 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:09.204 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190233280 0 0) 0x7ff138127440 con 0x7ff13814a700 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff118004eb0 con 0x7ff13814a700 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff1300014b0 con 0x7ff13814a700 2024-03-29T11:15:09.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3257778739 0 0) 0x7ff138127f90 con 0x7ff1380b0770 2024-03-29T11:15:09.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 >> v1:172.21.15.105:6790/0 conn(0x7ff13814a700 legacy=0x7ff1381e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 >> v1:172.21.15.110:6789/0 conn(0x7ff1380ac5a0 legacy=0x7ff138144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 monclient: found mon.a 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138145b70 con 0x7ff1380b0770 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff12f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 10 monclient: _renew_subs 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138126f10 con 0x7ff1380b0770 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:09.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 10 monclient: _renew_subs 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.218+0000 7ff13cf3a640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1381262d0 con 0x7ff1380b0770 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.219+0000 7ff13cf3a640 1 librados: init done 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1200035f0 con 0x7ff1380b0770 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff13559a640 4 set_mon_vals no callback set 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff120003980 con 0x7ff1380b0770 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12f7fe640 10 monclient: dump: 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:09.208 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.220+0000 7ff12cd99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.221+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff120003b60 con 0x7ff1380b0770 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.221+0000 7ff12cd99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12cd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff1200847f0 con 0x7ff1380b0770 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12f7fe640 10 client.4461.objecter ms_dispatch 0x7ff138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12f7fe640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12f7fe640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12f7fe640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:15:09.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c005080 con 0x7ff1380b0770 2024-03-29T11:15:09.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.222+0000 7ff13659c640 10 monclient: get_auth_request con 0x7ff11806e0e0 auth_method 0 2024-03-29T11:15:09.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.223+0000 7ff12f7fe640 10 client.4461.objecter ms_handle_connect 0x7ff11806e0e0 2024-03-29T11:15:09.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.230+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff120050130 con 0x7ff1380b0770 2024-03-29T11:15:09.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.230+0000 7ff12f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:09.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.230+0000 7ff12f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:09.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.615+0000 7ff12cd99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.615+0000 7ff12cd99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.615+0000 7ff12cd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:09.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.615+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff11c005330 con 0x7ff1380b0770 2024-03-29T11:15:09.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.616+0000 7ff12f7fe640 1 -- 172.21.15.105:0/106913354 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff120083ea0 con 0x7ff1380b0770 2024-03-29T11:15:09.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.616+0000 7ff12f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:09.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.616+0000 7ff12f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:09.595 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:09.595 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.620+0000 7ff12cd99640 10 librados: watch_flush enter 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.620+0000 7ff12cd99640 10 librados: watch_flush exit 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.620+0000 7ff12cd99640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.620+0000 7ff12cd99640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff11806e0e0 legacy=0x7ff1180705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 10 monclient: shutdown 2024-03-29T11:15:09.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 >> v1:172.21.15.105:6789/0 conn(0x7ff1380b0770 legacy=0x7ff1381db8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 shutdown_connections 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 >> 172.21.15.105:0/106913354 conn(0x7ff13806b660 msgr2=0x7ff1380b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 shutdown_connections 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 -- 172.21.15.105:0/106913354 wait complete. 2024-03-29T11:15:09.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:09.621+0000 7ff12cd99640 1 librados: shutdown 2024-03-29T11:15:09.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:16.420 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.576+0000 7ff5df96b640 10 monclient: get_monmap_and_config 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.576+0000 7ff5df96b640 10 monclient: build_initial_monmap 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.576+0000 7ff5df96b640 10 monclient: monmap: 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:16.577125+0000 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:16.577125+0000 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:16.554 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:16.555 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:16.555 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.576+0000 7ff5df96b640 1 Processor -- start 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 1 -- start start 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient: init 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-c con 0x7ff5d81457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-b con 0x7ff5d813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-a con 0x7ff5d80a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d80aaef0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d8145bd0 con 0x7ff5d81457f0 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d8145eb0 con 0x7ff5d813f650 2024-03-29T11:15:16.557 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5df96b640 10 monclient(hunting): authenticate will time out at 1190.853743s 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5ddf08640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7ff5d80a9d70 0x7ff5d814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:38548/0 (socket says 172.21.15.105:38548) 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.577+0000 7ff5ddf08640 1 -- 172.21.15.105:0/1254273082 learned_addr learned my addr 172.21.15.105:0/1254273082 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349963172 0 0) 0x7ff5d80aaef0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.578+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4003520 con 0x7ff5d80a9d70 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285895550 0 0) 0x7ff5d8145bd0 con 0x7ff5d81457f0 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:15:16.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5d80aaef0 con 0x7ff5d81457f0 2024-03-29T11:15:16.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533569007 0 0) 0x7ff5c4003520 con 0x7ff5d80a9d70 2024-03-29T11:15:16.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.579+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 >> v1:172.21.15.105:6790/0 conn(0x7ff5d813f650 legacy=0x7ff5d80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 >> v1:172.21.15.110:6789/0 conn(0x7ff5d81457f0 legacy=0x7ff5d813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 1 monclient: found mon.noname-a 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.580+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8146340 con 0x7ff5d80a9d70 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:16.561 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5d0001990 con 0x7ff5d80a9d70 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: dump: 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:16.562 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5df96b640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5df96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d0001af0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 1 -- 172.21.15.105:0/1254273082 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5d0001f40 con 0x7ff5d80a9d70 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5cffff640 10 monclient: dump: 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.563 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5df96b640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.581+0000 7ff5df96b640 4 set_mon_vals no callback set 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.582+0000 7ff5df96b640 10 monclient: shutdown 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.582+0000 7ff5df96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.582+0000 7ff5df96b640 1 -- 172.21.15.105:0/1254273082 >> v1:172.21.15.105:6789/0 conn(0x7ff5d80a9d70 legacy=0x7ff5d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.583+0000 7ff5df96b640 1 -- 172.21.15.105:0/1254273082 shutdown_connections 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.583+0000 7ff5df96b640 1 -- 172.21.15.105:0/1254273082 >> 172.21.15.105:0/1254273082 conn(0x7ff5d8066a00 msgr2=0x7ff5d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.583+0000 7ff5df96b640 1 -- 172.21.15.105:0/1254273082 shutdown_connections 2024-03-29T11:15:16.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.583+0000 7ff5df96b640 1 -- 172.21.15.105:0/1254273082 wait complete. 2024-03-29T11:15:16.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.584+0000 7ff5df96b640 10 monclient: build_initial_monmap 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.584+0000 7ff5df96b640 10 monclient: monmap: 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 librados: starting msgr at 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 librados: starting objecter 2024-03-29T11:15:16.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 Processor -- start 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 -- start start 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 librados: setting wanted keys 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 1 librados: calling monclient init 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.585+0000 7ff5df96b640 10 monclient: init 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.586+0000 7ff5df96b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.586+0000 7ff5df96b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.586+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-a con 0x7ff5d80a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.586+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-b con 0x7ff5d813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.586+0000 7ff5dcf06640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff5d813f650 0x7ff5d81dc0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:44872/0 (socket says 172.21.15.105:44872) 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 10 monclient(hunting): picked mon.noname-c con 0x7ff5d81457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5dcf06640 1 -- 172.21.15.105:0/1419451839 learned_addr learned my addr 172.21.15.105:0/1419451839 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d81dbc50 con 0x7ff5d80a9d70 2024-03-29T11:15:16.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d81db730 con 0x7ff5d813f650 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d81db8c0 con 0x7ff5d81457f0 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 client.?.objecter ms_handle_connect 0x7ff5d813f650 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 client.?.objecter ms_handle_connect 0x7ff5d80a9d70 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:16.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5df96b640 10 monclient(hunting): authenticate will time out at 1190.863113s 2024-03-29T11:15:16.573 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948099513 0 0) 0x7ff5d81dbc50 con 0x7ff5d80a9d70 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5bc004eb0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361611596 0 0) 0x7ff5bc004eb0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5bc005300 con 0x7ff5d80a9d70 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5d4001870 con 0x7ff5d80a9d70 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:16.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): dump: 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:16.575 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.587+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 476453681 0 0) 0x7ff5bc005300 con 0x7ff5d80a9d70 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.588+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 >> v1:172.21.15.105:6790/0 conn(0x7ff5d81457f0 legacy=0x7ff5d813fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 >> v1:172.21.15.110:6789/0 conn(0x7ff5d813f650 legacy=0x7ff5d81dc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 1 monclient: found mon.a 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8142190 con 0x7ff5d80a9d70 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.590+0000 7ff5df96b640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:15:16.576 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 10 monclient: _renew_subs 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d8142380 con 0x7ff5d80a9d70 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 10 monclient: _renew_subs 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d81e73d0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5df96b640 1 librados: init done 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d40035f0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.591+0000 7ff5cf7fe640 4 set_mon_vals no callback set 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.592+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5d4001870 con 0x7ff5d80a9d70 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.592+0000 7ff5cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.592+0000 7ff5cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.592+0000 7ff5cdffb640 10 monclient: dump: 2024-03-29T11:15:16.577 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:16.578 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.592+0000 7ff5df96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.593+0000 7ff5df96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.593+0000 7ff5df96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.593+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b8005080 con 0x7ff5d80a9d70 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.594+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff5d40195d0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff5d4083280 con 0x7ff5d80a9d70 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 10 client.4467.objecter ms_dispatch 0x7ff5d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:16.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:15:16.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5dcf06640 10 monclient: get_auth_request con 0x7ff5bc06e180 auth_method 0 2024-03-29T11:15:16.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.595+0000 7ff5cdffb640 10 client.4467.objecter ms_handle_connect 0x7ff5bc06e180 2024-03-29T11:15:16.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.601+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5d404ebc0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.601+0000 7ff5cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:16.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.601+0000 7ff5cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:16.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.894+0000 7ff5df96b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.894+0000 7ff5df96b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.894+0000 7ff5df96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:16.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.894+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b8005330 con 0x7ff5d80a9d70 2024-03-29T11:15:16.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.895+0000 7ff5cdffb640 1 -- 172.21.15.105:0/1419451839 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff5d404ebc0 con 0x7ff5d80a9d70 2024-03-29T11:15:16.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.895+0000 7ff5cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.895+0000 7ff5cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:16.873 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:16.873 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:16.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.898+0000 7ff5df96b640 10 librados: watch_flush enter 2024-03-29T11:15:16.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.898+0000 7ff5df96b640 10 librados: watch_flush exit 2024-03-29T11:15:16.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.898+0000 7ff5df96b640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:15:16.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.898+0000 7ff5df96b640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:15:16.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff5bc06e180 legacy=0x7ff5bc070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 10 monclient: shutdown 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 >> v1:172.21.15.105:6789/0 conn(0x7ff5d80a9d70 legacy=0x7ff5d806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 shutdown_connections 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 >> 172.21.15.105:0/1419451839 conn(0x7ff5d8066a00 msgr2=0x7ff5d80b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 shutdown_connections 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 -- 172.21.15.105:0/1419451839 wait complete. 2024-03-29T11:15:16.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:16.899+0000 7ff5df96b640 1 librados: shutdown 2024-03-29T11:15:16.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:23.698 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.862+0000 7fcb5ab5d640 10 monclient: get_monmap_and_config 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.862+0000 7fcb5ab5d640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.862+0000 7fcb5ab5d640 10 monclient: monmap: 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:23.863481+0000 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:23.863481+0000 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.840 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.863+0000 7fcb5ab5d640 1 Processor -- start 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.863+0000 7fcb5ab5d640 1 -- start start 2024-03-29T11:15:23.841 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.863+0000 7fcb5ab5d640 10 monclient: init 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb541457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb540a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb53fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fcb5413f650 0x7fcb540b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:45434/0 (socket says 172.21.15.105:45434) 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb53fff640 1 -- 172.21.15.105:0/1242635868 learned_addr learned my addr 172.21.15.105:0/1242635868 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:23.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.865+0000 7fcb5ab5d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb540aaef0 con 0x7fcb541457f0 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb54145bd0 con 0x7fcb540a9d70 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb54145eb0 con 0x7fcb5413f650 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb5ab5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb5ab5d640 10 monclient(hunting): authenticate will time out at 1198.142116s 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343678203 0 0) 0x7fcb540aaef0 con 0x7fcb541457f0 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:15:23.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.866+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb44003520 con 0x7fcb541457f0 2024-03-29T11:15:23.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4290407406 0 0) 0x7fcb44003520 con 0x7fcb541457f0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 >> v1:172.21.15.105:6790/0 conn(0x7fcb5413f650 legacy=0x7fcb540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 >> v1:172.21.15.110:6789/0 conn(0x7fcb540a9d70 legacy=0x7fcb5414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 1 monclient: found mon.noname-a 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.867+0000 7fcb537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54146340 con 0x7fcb541457f0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcb48001690 con 0x7fcb541457f0 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:23.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: dump: 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb5ab5d640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb5ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001920 con 0x7fcb541457f0 2024-03-29T11:15:23.849 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.868+0000 7fcb537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb5ab5d640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb5ab5d640 4 set_mon_vals no callback set 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb537fe640 1 -- 172.21.15.105:0/1242635868 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcb48001cb0 con 0x7fcb541457f0 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb537fe640 10 monclient: dump: 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.850 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb5ab5d640 10 monclient: shutdown 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb5ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.869+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 >> v1:172.21.15.105:6789/0 conn(0x7fcb541457f0 legacy=0x7fcb5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.870+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 shutdown_connections 2024-03-29T11:15:23.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.870+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 >> 172.21.15.105:0/1242635868 conn(0x7fcb54066a00 msgr2=0x7fcb54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:23.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.870+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 shutdown_connections 2024-03-29T11:15:23.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.870+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1242635868 wait complete. 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.871+0000 7fcb5ab5d640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.871+0000 7fcb5ab5d640 10 monclient: monmap: 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 librados: starting msgr at 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 librados: starting objecter 2024-03-29T11:15:23.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 Processor -- start 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 -- start start 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 librados: setting wanted keys 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 1 librados: calling monclient init 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.872+0000 7fcb5ab5d640 10 monclient: init 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb540a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb588f9640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fcb540a9d70 0x7fcb54142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:60892/0 (socket says 172.21.15.105:60892) 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb588f9640 1 -- 172.21.15.105:0/1932808594 learned_addr learned my addr 172.21.15.105:0/1932808594 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb541457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb517fa640 10 client.?.objecter ms_handle_connect 0x7fcb540a9d70 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb517fa640 10 client.?.objecter ms_handle_connect 0x7fcb5413f650 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.873+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb540ac500 con 0x7fcb540a9d70 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb54142340 con 0x7fcb541457f0 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153600172 0 0) 0x7fcb540ac500 con 0x7fcb540a9d70 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb54142010 con 0x7fcb5413f650 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb5ab5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb5ab5d640 10 monclient(hunting): authenticate will time out at 1198.150135s 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb38004eb0 con 0x7fcb540a9d70 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351765170 0 0) 0x7fcb54142010 con 0x7fcb5413f650 2024-03-29T11:15:23.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb540ac500 con 0x7fcb5413f650 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242877662 0 0) 0x7fcb38004eb0 con 0x7fcb540a9d70 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb38005490 con 0x7fcb540a9d70 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcb48001690 con 0x7fcb540a9d70 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): dump: 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.862 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897727819 0 0) 0x7fcb540ac500 con 0x7fcb5413f650 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb38004eb0 con 0x7fcb5413f650 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1896031754 0 0) 0x7fcb38005490 con 0x7fcb540a9d70 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 >> v1:172.21.15.105:6790/0 conn(0x7fcb5413f650 legacy=0x7fcb54141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.874+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 >> v1:172.21.15.110:6789/0 conn(0x7fcb541457f0 legacy=0x7fcb54136f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 1 monclient: found mon.a 2024-03-29T11:15:23.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb541448c0 con 0x7fcb540a9d70 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 10 monclient: _renew_subs 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb54144b70 con 0x7fcb540a9d70 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 10 monclient: _renew_subs 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb54137830 con 0x7fcb540a9d70 2024-03-29T11:15:23.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb5ab5d640 1 librados: init done 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001270 con 0x7fcb540a9d70 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fcb48003b00 con 0x7fcb540a9d70 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.876+0000 7fcb517fa640 10 monclient: dump: 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:23.865 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.878+0000 7fcb5ab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.878+0000 7fcb5ab5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.878+0000 7fcb5ab5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.878+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb54075f10 con 0x7fcb540a9d70 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.878+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fcb48003c60 con 0x7fcb540a9d70 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb527fc640 4 set_mon_vals no callback set 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fcb48084ae0 con 0x7fcb540a9d70 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb517fa640 10 client.4473.objecter ms_dispatch 0x7fcb54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb517fa640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb517fa640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.879+0000 7fcb517fa640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.880+0000 7fcb53fff640 10 monclient: get_auth_request con 0x7fcb3806df00 auth_method 0 2024-03-29T11:15:23.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.881+0000 7fcb517fa640 10 client.4473.objecter ms_handle_connect 0x7fcb3806df00 2024-03-29T11:15:23.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.887+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb48050420 con 0x7fcb540a9d70 2024-03-29T11:15:23.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.887+0000 7fcb517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:23.887+0000 7fcb517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:24.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.211+0000 7fcb5ab5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.211+0000 7fcb5ab5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.211+0000 7fcb5ab5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:24.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.211+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb540b1810 con 0x7fcb540a9d70 2024-03-29T11:15:24.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.214+0000 7fcb517fa640 1 -- 172.21.15.105:0/1932808594 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fcb48052350 con 0x7fcb540a9d70 2024-03-29T11:15:24.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.214+0000 7fcb517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:24.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.214+0000 7fcb517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:24.191 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:24.192 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:24.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.217+0000 7fcb5ab5d640 10 librados: watch_flush enter 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 10 librados: watch_flush exit 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 >> v1:172.21.15.110:6816/1087970004 conn(0x7fcb3806df00 legacy=0x7fcb380703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 10 monclient: shutdown 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.218+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 >> v1:172.21.15.105:6789/0 conn(0x7fcb540a9d70 legacy=0x7fcb54142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.219+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 shutdown_connections 2024-03-29T11:15:24.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.219+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 >> 172.21.15.105:0/1932808594 conn(0x7fcb54066a00 msgr2=0x7fcb54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:24.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.219+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 shutdown_connections 2024-03-29T11:15:24.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.219+0000 7fcb5ab5d640 1 -- 172.21.15.105:0/1932808594 wait complete. 2024-03-29T11:15:24.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:24.219+0000 7fcb5ab5d640 1 librados: shutdown 2024-03-29T11:15:24.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:31.022 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.193+0000 7f0d39754640 10 monclient: get_monmap_and_config 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.193+0000 7f0d39754640 10 monclient: build_initial_monmap 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.193+0000 7f0d39754640 10 monclient: monmap: 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:31.194529+0000 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:31.194529+0000 2024-03-29T11:15:31.171 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.194+0000 7f0d39754640 1 Processor -- start 2024-03-29T11:15:31.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.194+0000 7f0d39754640 1 -- start start 2024-03-29T11:15:31.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.194+0000 7f0d39754640 10 monclient: init 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-a con 0x7f0d341457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3413f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-b con 0x7f0d340a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:31.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340aaef0 con 0x7f0d341457f0 2024-03-29T11:15:31.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d34145bd0 con 0x7f0d3413f650 2024-03-29T11:15:31.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d34145eb0 con 0x7f0d340a9d70 2024-03-29T11:15:31.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:31.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.195+0000 7f0d39754640 10 monclient(hunting): authenticate will time out at 1205.471336s 2024-03-29T11:15:31.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.198+0000 7f0d337fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f0d340a9d70 0x7f0d3414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60043/0 (socket says 172.21.15.105:60043) 2024-03-29T11:15:31.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.198+0000 7f0d337fe640 1 -- 172.21.15.105:0/705808350 learned_addr learned my addr 172.21.15.105:0/705808350 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498476390 0 0) 0x7f0d34145eb0 con 0x7f0d340a9d70 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1c003520 con 0x7f0d340a9d70 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.200+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890290405 0 0) 0x7f0d1c003520 con 0x7f0d340a9d70 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 >> v1:172.21.15.110:6789/0 conn(0x7f0d3413f650 legacy=0x7f0d340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 >> v1:172.21.15.105:6789/0 conn(0x7f0d341457f0 legacy=0x7f0d3413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:31.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 monclient: found mon.noname-b 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34146340 con 0x7f0d340a9d70 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d39754640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d39754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d2c001a10 con 0x7f0d340a9d70 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: dump: 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:31.184 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d39754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c001ca0 con 0x7f0d340a9d70 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d39754640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:31.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.201+0000 7f0d39754640 4 set_mon_vals no callback set 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d31ffb640 1 -- 172.21.15.105:0/705808350 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d2c002030 con 0x7f0d340a9d70 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d31ffb640 10 monclient: dump: 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:31.186 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 10 monclient: shutdown 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 1 -- 172.21.15.105:0/705808350 >> v1:172.21.15.105:6790/0 conn(0x7f0d340a9d70 legacy=0x7f0d3414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 1 -- 172.21.15.105:0/705808350 shutdown_connections 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 1 -- 172.21.15.105:0/705808350 >> 172.21.15.105:0/705808350 conn(0x7f0d34066a00 msgr2=0x7f0d34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.202+0000 7f0d39754640 1 -- 172.21.15.105:0/705808350 shutdown_connections 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.203+0000 7f0d39754640 1 -- 172.21.15.105:0/705808350 wait complete. 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.203+0000 7f0d39754640 10 monclient: build_initial_monmap 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.203+0000 7f0d39754640 10 monclient: monmap: 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:31.187 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 librados: starting msgr at 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 librados: starting objecter 2024-03-29T11:15:31.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 Processor -- start 2024-03-29T11:15:31.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 -- start start 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 librados: setting wanted keys 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 1 librados: calling monclient init 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.204+0000 7f0d39754640 10 monclient: init 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:31.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-c con 0x7f0d340a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3413f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient(hunting): picked mon.noname-b con 0x7f0d341457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d34135930 con 0x7f0d3413f650 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d341422d0 con 0x7f0d341457f0 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d34142460 con 0x7f0d340a9d70 2024-03-29T11:15:31.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.205+0000 7f0d39754640 10 monclient(hunting): authenticate will time out at 1205.481451s 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.208+0000 7f0d32ffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f0d340a9d70 0x7f0d341428d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60087/0 (socket says 172.21.15.105:60087) 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.208+0000 7f0d32ffd640 1 -- 172.21.15.105:0/2954110615 learned_addr learned my addr 172.21.15.105:0/2954110615 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.209+0000 7f0d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f0d340a9d70 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.209+0000 7f0d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:31.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942429867 0 0) 0x7f0d34142460 con 0x7f0d340a9d70 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.212+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20004eb0 con 0x7f0d340a9d70 2024-03-29T11:15:31.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f0d341457f0 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 543142049 0 0) 0x7f0d20004eb0 con 0x7f0d340a9d70 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d34142460 con 0x7f0d340a9d70 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d240017d0 con 0x7f0d340a9d70 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.206 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:31.207 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973582739 0 0) 0x7f0d341422d0 con 0x7f0d341457f0 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20004eb0 con 0x7f0d341457f0 2024-03-29T11:15:31.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1584028292 0 0) 0x7f0d34142460 con 0x7f0d340a9d70 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.213+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 >> v1:172.21.15.110:6789/0 conn(0x7f0d341457f0 legacy=0x7f0d341e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 >> v1:172.21.15.105:6789/0 conn(0x7f0d3413f650 legacy=0x7f0d3413fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 1 monclient: found mon.c 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:31.209 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34140c20 con 0x7f0d340a9d70 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.214+0000 7f0d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:15:31.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 10 monclient: _renew_subs 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3413b440 con 0x7f0d340a9d70 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 10 monclient: _renew_subs 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341e7400 con 0x7f0d340a9d70 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.215+0000 7f0d39754640 1 librados: init done 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.216+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d24001d60 con 0x7f0d340a9d70 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.216+0000 7f0d1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.216+0000 7f0d317fa640 4 set_mon_vals no callback set 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.216+0000 7f0d39754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.217+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0d24004a00 con 0x7f0d340a9d70 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.217+0000 7f0d39754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.217+0000 7f0d39754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.218+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3413b970 con 0x7f0d340a9d70 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.218+0000 7f0d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.218+0000 7f0d1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:31.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.218+0000 7f0d1b7fe640 10 monclient: dump: 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:31.214 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.219+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f0d2401aa50 con 0x7f0d340a9d70 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.221+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f0d24054880 con 0x7f0d340a9d70 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d1b7fe640 10 client.4544.objecter ms_dispatch 0x7f0d34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d1b7fe640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d1b7fe640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d1b7fe640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d327fc640 10 monclient: get_auth_request con 0x7f0d2006dd90 auth_method 0 2024-03-29T11:15:31.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.222+0000 7f0d1b7fe640 10 client.4544.objecter ms_handle_connect 0x7f0d2006dd90 2024-03-29T11:15:31.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.232+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d24050b20 con 0x7f0d340a9d70 2024-03-29T11:15:31.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.232+0000 7f0d1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:31.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.232+0000 7f0d1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:31.527 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.549+0000 7f0d39754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.549+0000 7f0d39754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.550+0000 7f0d39754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:31.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.550+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d341425f0 con 0x7f0d340a9d70 2024-03-29T11:15:31.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.550+0000 7f0d1b7fe640 1 -- 172.21.15.105:0/2954110615 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f0d341425f0 con 0x7f0d340a9d70 2024-03-29T11:15:31.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.550+0000 7f0d1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:31.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.550+0000 7f0d1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:31.531 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:31.531 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.555+0000 7f0d39754640 10 librados: watch_flush enter 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 10 librados: watch_flush exit 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 >> v1:172.21.15.110:6816/1087970004 conn(0x7f0d2006dd90 legacy=0x7f0d20070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 10 monclient: shutdown 2024-03-29T11:15:31.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:31.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 >> v1:172.21.15.105:6790/0 conn(0x7f0d340a9d70 legacy=0x7f0d341428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:31.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 shutdown_connections 2024-03-29T11:15:31.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 >> 172.21.15.105:0/2954110615 conn(0x7f0d34066a00 msgr2=0x7f0d340b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:31.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 shutdown_connections 2024-03-29T11:15:31.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.558+0000 7f0d39754640 1 -- 172.21.15.105:0/2954110615 wait complete. 2024-03-29T11:15:31.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:31.559+0000 7f0d39754640 1 librados: shutdown 2024-03-29T11:15:31.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:38.364 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:38.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.537+0000 7f33fe554640 10 monclient: get_monmap_and_config 2024-03-29T11:15:38.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.538+0000 7f33fe554640 10 monclient: build_initial_monmap 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.538+0000 7f33fe554640 10 monclient: monmap: 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:38.539068+0000 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:38.539068+0000 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:38.516 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.517 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.540+0000 7f33fe554640 1 Processor -- start 2024-03-29T11:15:38.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.540+0000 7f33fe554640 1 -- start start 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.540+0000 7f33fe554640 10 monclient: init 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-b con 0x7f33f81457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-c con 0x7f33f813f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-a con 0x7f33f80a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80aaef0 con 0x7f33f80a9d70 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f8145bd0 con 0x7f33f813f650 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f8145eb0 con 0x7f33f81457f0 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:38.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.541+0000 7f33fe554640 10 monclient(hunting): authenticate will time out at 1212.817515s 2024-03-29T11:15:38.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.546+0000 7f33f77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f33f813f650 0x7f33f80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:47257/0 (socket says 172.21.15.105:47257) 2024-03-29T11:15:38.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.546+0000 7f33f77fe640 1 -- 172.21.15.105:0/3267739932 learned_addr learned my addr 172.21.15.105:0/3267739932 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:38.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435458551 0 0) 0x7f33f8145bd0 con 0x7f33f813f650 2024-03-29T11:15:38.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e0003520 con 0x7f33f813f650 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177340351 0 0) 0x7f33e0003520 con 0x7f33f813f650 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 >> v1:172.21.15.105:6790/0 conn(0x7f33f81457f0 legacy=0x7f33f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.549+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 >> v1:172.21.15.105:6789/0 conn(0x7f33f80a9d70 legacy=0x7f33f814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8146340 con 0x7f33f813f650 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:38.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f33f0001a10 con 0x7f33f813f650 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33f6ffd640 10 monclient: dump: 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33fe554640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:15:38.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.550+0000 7f33fe554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:38.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f0001be0 con 0x7f33f813f650 2024-03-29T11:15:38.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 1 -- 172.21.15.105:0/3267739932 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f33f0002030 con 0x7f33f813f650 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33f6ffd640 10 monclient: dump: 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33fe554640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:38.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.553+0000 7f33fe554640 4 set_mon_vals no callback set 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.554+0000 7f33fe554640 10 monclient: shutdown 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.554+0000 7f33fe554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.554+0000 7f33fe554640 1 -- 172.21.15.105:0/3267739932 >> v1:172.21.15.110:6789/0 conn(0x7f33f813f650 legacy=0x7f33f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.555+0000 7f33fe554640 1 -- 172.21.15.105:0/3267739932 shutdown_connections 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.555+0000 7f33fe554640 1 -- 172.21.15.105:0/3267739932 >> 172.21.15.105:0/3267739932 conn(0x7f33f8066a00 msgr2=0x7f33f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.555+0000 7f33fe554640 1 -- 172.21.15.105:0/3267739932 shutdown_connections 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.555+0000 7f33fe554640 1 -- 172.21.15.105:0/3267739932 wait complete. 2024-03-29T11:15:38.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.556+0000 7f33fe554640 10 monclient: build_initial_monmap 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.556+0000 7f33fe554640 10 monclient: monmap: 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.556+0000 7f33fe554640 1 librados: starting msgr at 2024-03-29T11:15:38.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.556+0000 7f33fe554640 1 librados: starting objecter 2024-03-29T11:15:38.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.556+0000 7f33fe554640 1 Processor -- start 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 1 -- start start 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 1 librados: setting wanted keys 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 1 librados: calling monclient init 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 10 monclient: init 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.557+0000 7f33fe554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-b con 0x7f33f80a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-c con 0x7f33f813f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 10 monclient(hunting): picked mon.noname-a con 0x7f33f81457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f81dfb60 con 0x7f33f81457f0 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f806f480 con 0x7f33f80a9d70 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f8137d20 con 0x7f33f813f650 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:38.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.559+0000 7f33fe554640 10 monclient(hunting): authenticate will time out at 1212.835491s 2024-03-29T11:15:38.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f77fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f33f813f650 0x7f33f81ded40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:45595/0 (socket says 172.21.15.105:45595) 2024-03-29T11:15:38.546 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f77fe640 1 -- 172.21.15.105:0/1936214213 learned_addr learned my addr 172.21.15.105:0/1936214213 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 client.?.objecter ms_handle_connect 0x7f33f813f650 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 client.?.objecter ms_handle_connect 0x7f33f80a9d70 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866554928 0 0) 0x7f33f8137d20 con 0x7f33f813f650 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.547 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4004eb0 con 0x7f33f813f650 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.563+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204462127 0 0) 0x7f33f806f480 con 0x7f33f80a9d70 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:15:38.548 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f8137d20 con 0x7f33f80a9d70 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 client.?.objecter ms_handle_connect 0x7f33f81457f0 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011749399 0 0) 0x7f33e4004eb0 con 0x7f33f813f650 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f806f480 con 0x7f33f813f650 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f33f0001ef0 con 0x7f33f813f650 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:38.549 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:38.550 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808920423 0 0) 0x7f33f81dfb60 con 0x7f33f81457f0 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:15:38.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4004eb0 con 0x7f33f81457f0 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 545460436 0 0) 0x7f33f806f480 con 0x7f33f813f650 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 >> v1:172.21.15.110:6789/0 conn(0x7f33f80a9d70 legacy=0x7f33f806f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 >> v1:172.21.15.105:6789/0 conn(0x7f33f81457f0 legacy=0x7f33f81df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 monclient: found mon.c 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8137eb0 con 0x7f33f813f650 2024-03-29T11:15:38.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 10 monclient: _renew_subs 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f8142430 con 0x7f33f813f650 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 10 monclient: _renew_subs 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f8144f70 con 0x7f33f813f650 2024-03-29T11:15:38.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.564+0000 7f33fe554640 1 librados: init done 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f0004fe0 con 0x7f33f813f650 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f33f00053b0 con 0x7f33f813f650 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.566+0000 7f33f4ff9640 10 monclient: dump: 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:38.554 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.565+0000 7f33f67fc640 4 set_mon_vals no callback set 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.566+0000 7f33de7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.567+0000 7f33de7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.567+0000 7f33de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.567+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f806f280 con 0x7f33f813f650 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.567+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f33f0005510 con 0x7f33f813f650 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f7fff640 10 monclient: get_auth_request con 0x7f33e406df70 auth_method 0 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f33f0084f80 con 0x7f33f813f650 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 10 client.4547.objecter ms_dispatch 0x7f33f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:15:38.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.568+0000 7f33f4ff9640 10 client.4547.objecter ms_handle_connect 0x7f33e406df70 2024-03-29T11:15:38.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.575+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33f0050860 con 0x7f33f813f650 2024-03-29T11:15:38.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.575+0000 7f33f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:38.556 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.575+0000 7f33f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:38.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.906+0000 7f33de7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.906+0000 7f33de7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.906+0000 7f33de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.906+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33f8142780 con 0x7f33f813f650 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.907+0000 7f33f4ff9640 1 -- 172.21.15.105:0/1936214213 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f33f0052790 con 0x7f33f813f650 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.907+0000 7f33f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.907+0000 7f33f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:38.885 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:38.886 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 10 librados: watch_flush enter 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 10 librados: watch_flush exit 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 >> v1:172.21.15.110:6816/1087970004 conn(0x7f33e406df70 legacy=0x7f33e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 10 monclient: shutdown 2024-03-29T11:15:38.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:38.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.910+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 >> v1:172.21.15.105:6790/0 conn(0x7f33f813f650 legacy=0x7f33f81ded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.911+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 shutdown_connections 2024-03-29T11:15:38.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.911+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 >> 172.21.15.105:0/1936214213 conn(0x7f33f8066a00 msgr2=0x7f33f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:38.892 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.913+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 shutdown_connections 2024-03-29T11:15:38.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.916+0000 7f33de7fc640 1 -- 172.21.15.105:0/1936214213 wait complete. 2024-03-29T11:15:38.893 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:38.916+0000 7f33de7fc640 1 librados: shutdown 2024-03-29T11:15:38.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:45.719 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:45.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.893+0000 7ff5b1a84640 10 monclient: get_monmap_and_config 2024-03-29T11:15:45.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.893+0000 7ff5b1a84640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.893+0000 7ff5b1a84640 10 monclient: monmap: 2024-03-29T11:15:45.871 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:45.871 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:45.894921+0000 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:45.894921+0000 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.895+0000 7ff5b1a84640 1 Processor -- start 2024-03-29T11:15:45.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.895+0000 7ff5b1a84640 1 -- start start 2024-03-29T11:15:45.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.895+0000 7ff5b1a84640 10 monclient: init 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ac0b4920 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ac0b4200 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ac13f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0b0930 con 0x7ff5ac0b4200 2024-03-29T11:15:45.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0af460 con 0x7ff5ac0b4920 2024-03-29T11:15:45.875 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0aeda0 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.875 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.875 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.896+0000 7ff5b1a84640 10 monclient(hunting): authenticate will time out at 1220.172325s 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5b1283640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7ff5ac13f9a0 0x7ff5ac14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:48399/0 (socket says 172.21.15.105:48399) 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5b1283640 1 -- 172.21.15.105:0/896315366 learned_addr learned my addr 172.21.15.105:0/896315366 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944923282 0 0) 0x7ff5ac0aeda0 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:15:45.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.899+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff594003520 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.882 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217493745 0 0) 0x7ff5ac0b0930 con 0x7ff5ac0b4200 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ac0aeda0 con 0x7ff5ac0b4200 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466058134 0 0) 0x7ff594003520 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 >> v1:172.21.15.110:6789/0 conn(0x7ff5ac0b4920 legacy=0x7ff5ac13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 >> v1:172.21.15.105:6789/0 conn(0x7ff5ac0b4200 legacy=0x7ff5ac0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 monclient: found mon.noname-b 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:15:45.883 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0af980 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5a4001a10 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5ab7fe640 10 monclient: dump: 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:45.884 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5b1a84640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.903+0000 7ff5b1a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.904+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5a4001b70 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.904+0000 7ff5ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.904+0000 7ff5b1a84640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.904+0000 7ff5b1a84640 4 set_mon_vals no callback set 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5ab7fe640 1 -- 172.21.15.105:0/896315366 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff5a4001fc0 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5ab7fe640 10 monclient: dump: 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:45.885 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:45.886 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5b1a84640 10 monclient: shutdown 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5b1a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5b1a84640 1 -- 172.21.15.105:0/896315366 >> v1:172.21.15.105:6790/0 conn(0x7ff5ac13f9a0 legacy=0x7ff5ac14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5b1a84640 1 -- 172.21.15.105:0/896315366 shutdown_connections 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.905+0000 7ff5b1a84640 1 -- 172.21.15.105:0/896315366 >> 172.21.15.105:0/896315366 conn(0x7ff5ac06d230 msgr2=0x7ff5ac066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.908+0000 7ff5b1a84640 1 -- 172.21.15.105:0/896315366 shutdown_connections 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.908+0000 7ff5b1a84640 1 -- 172.21.15.105:0/896315366 wait complete. 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.909+0000 7ff5b1a84640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.909+0000 7ff5b1a84640 10 monclient: monmap: 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:45.888 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 librados: starting msgr at 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 librados: starting objecter 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 Processor -- start 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 -- start start 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 librados: setting wanted keys 2024-03-29T11:15:45.889 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 1 librados: calling monclient init 2024-03-29T11:15:45.890 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.910+0000 7ff5b1a84640 10 monclient: init 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ac0b4200 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ac0b4920 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ac13f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0a9bd0 con 0x7ff5ac0b4920 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0ae710 con 0x7ff5ac0b4200 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0acc10 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.891 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.911+0000 7ff5b1a84640 10 monclient(hunting): authenticate will time out at 1220.187516s 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5b0a82640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff5ac0b4200 0x7ff5ac0ac500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:51425/0 (socket says 172.21.15.105:51425) 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5b0a82640 1 -- 172.21.15.105:0/293695275 learned_addr learned my addr 172.21.15.105:0/293695275 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 client.?.objecter ms_handle_connect 0x7ff5ac0b4200 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522171329 0 0) 0x7ff5ac0ae710 con 0x7ff5ac0b4200 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.896 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.915+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff598004eb0 con 0x7ff5ac0b4200 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1256189381 0 0) 0x7ff598004eb0 con 0x7ff5ac0b4200 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5ac0ae710 con 0x7ff5ac0b4200 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff59c004080 con 0x7ff5ac0b4200 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): dump: 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:45.897 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 client.?.objecter ms_handle_connect 0x7ff5ac13f9a0 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899116163 0 0) 0x7ff5ac0acc10 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.916+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff598004eb0 con 0x7ff5ac13f9a0 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082010979 0 0) 0x7ff5ac0ae710 con 0x7ff5ac0b4200 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 >> v1:172.21.15.105:6790/0 conn(0x7ff5ac13f9a0 legacy=0x7ff5ac1db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.898 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 >> v1:172.21.15.105:6789/0 conn(0x7ff5ac0b4920 legacy=0x7ff5ac13a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 1 monclient: found mon.b 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0ad8f0 con 0x7ff5ac0b4200 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5b1a84640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5b1a84640 10 monclient: _renew_subs 2024-03-29T11:15:45.899 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5b1a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.917+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac140000 con 0x7ff5ac0b4200 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.918+0000 7ff5b1a84640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.918+0000 7ff5b1a84640 10 monclient: _renew_subs 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.918+0000 7ff5b1a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.918+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac13aec0 con 0x7ff5ac0b4200 2024-03-29T11:15:45.900 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.918+0000 7ff5b1a84640 1 librados: init done 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff59c004420 con 0x7ff5ac0b4200 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff59c001320 con 0x7ff5ac0b4200 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5a97fa640 10 monclient: dump: 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:45.901 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5b1a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5b1a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5b1a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.919+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac04f730 con 0x7ff5ac0b4200 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff59c001500 con 0x7ff5ac0b4200 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff59c054790 con 0x7ff5ac0b4200 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 10 client.4483.objecter ms_dispatch 0x7ff5ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:45.902 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.921+0000 7ff5a97fa640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:15:45.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.920+0000 7ff5aa7fc640 4 set_mon_vals no callback set 2024-03-29T11:15:45.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.922+0000 7ff5abfff640 10 monclient: get_auth_request con 0x7ff59806df60 auth_method 0 2024-03-29T11:15:45.903 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.923+0000 7ff5a97fa640 10 client.4483.objecter ms_handle_connect 0x7ff59806df60 2024-03-29T11:15:45.907 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.930+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff59c050a30 con 0x7ff5ac0b4200 2024-03-29T11:15:45.908 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.930+0000 7ff5a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.908 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:45.930+0000 7ff5a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:46.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.248+0000 7ff5b1a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.248+0000 7ff5b1a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.248+0000 7ff5b1a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:15:46.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.248+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5ac0b2180 con 0x7ff5ac0b4200 2024-03-29T11:15:46.229 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:46.229 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:46.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.251+0000 7ff5a97fa640 1 -- 172.21.15.105:0/293695275 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff59c050a30 con 0x7ff5ac0b4200 2024-03-29T11:15:46.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.251+0000 7ff5a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:46.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.251+0000 7ff5a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 10 librados: watch_flush enter 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 10 librados: watch_flush exit 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff59806df60 legacy=0x7ff598070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 10 monclient: shutdown 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:46.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.254+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 >> v1:172.21.15.110:6789/0 conn(0x7ff5ac0b4200 legacy=0x7ff5ac0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.256+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 shutdown_connections 2024-03-29T11:15:46.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.256+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 >> 172.21.15.105:0/293695275 conn(0x7ff5ac06d230 msgr2=0x7ff5ac0b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:46.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.256+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 shutdown_connections 2024-03-29T11:15:46.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.256+0000 7ff5b1a84640 1 -- 172.21.15.105:0/293695275 wait complete. 2024-03-29T11:15:46.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:46.256+0000 7ff5b1a84640 1 librados: shutdown 2024-03-29T11:15:46.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:53.060 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:53.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.230+0000 7f1250df6640 10 monclient: get_monmap_and_config 2024-03-29T11:15:53.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.230+0000 7f1250df6640 10 monclient: build_initial_monmap 2024-03-29T11:15:53.208 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.230+0000 7f1250df6640 10 monclient: monmap: 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:15:53.231703+0000 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:15:53.231703+0000 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.209 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:53.210 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:15:53.210 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:15:53.210 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.231+0000 7f1250df6640 1 Processor -- start 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.231+0000 7f1250df6640 1 -- start start 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.231+0000 7f1250df6640 10 monclient: init 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-a con 0x7f124c0b4790 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-b con 0x7f124c0aea90 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-c con 0x7f124c0b39f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0b0c60 con 0x7f124c0b4790 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0af020 con 0x7f124c0b39f0 2024-03-29T11:15:53.221 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c13fbd0 con 0x7f124c0aea90 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.232+0000 7f1250df6640 10 monclient(hunting): authenticate will time out at 1227.508849s 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124a59c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f124c0b4790 0x7f124c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:54984/0 (socket says 172.21.15.105:54984) 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124a59c640 1 -- 172.21.15.105:0/1626628597 learned_addr learned my addr 172.21.15.105:0/1626628597 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519917074 0 0) 0x7f124c0b0c60 con 0x7f124c0b4790 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1234003520 con 0x7f124c0b4790 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465711204 0 0) 0x7f124c13fbd0 con 0x7f124c0aea90 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.233+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c0b0c60 con 0x7f124c0aea90 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1348547620 0 0) 0x7f1234003520 con 0x7f124c0b4790 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 >> v1:172.21.15.105:6790/0 conn(0x7f124c0aea90 legacy=0x7f124c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 >> v1:172.21.15.110:6789/0 conn(0x7f124c0b39f0 legacy=0x7f124c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 monclient: found mon.noname-a 2024-03-29T11:15:53.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c13fd60 con 0x7f124c0b4790 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: _finish_auth 0 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f1250df6640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:15:53.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f1250df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f123c001690 con 0x7f124c0b4790 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: dump: 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.225 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:53.226 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f1250df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f123c0018b0 con 0x7f124c0b4790 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 1 -- 172.21.15.105:0/1626628597 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f123c001c40 con 0x7f124c0b4790 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f1250df6640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:53.227 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f1250df6640 4 set_mon_vals no callback set 2024-03-29T11:15:53.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:53.228 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.234+0000 7f124959a640 10 monclient: dump: 2024-03-29T11:15:53.228 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:53.228 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.229 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.229 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.229 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 10 monclient: shutdown 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 1 -- 172.21.15.105:0/1626628597 >> v1:172.21.15.105:6789/0 conn(0x7f124c0b4790 legacy=0x7f124c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.230 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 1 -- 172.21.15.105:0/1626628597 shutdown_connections 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 1 -- 172.21.15.105:0/1626628597 >> 172.21.15.105:0/1626628597 conn(0x7f124c066a00 msgr2=0x7f124c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 1 -- 172.21.15.105:0/1626628597 shutdown_connections 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.235+0000 7f1250df6640 1 -- 172.21.15.105:0/1626628597 wait complete. 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.236+0000 7f1250df6640 10 monclient: build_initial_monmap 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.236+0000 7f1250df6640 10 monclient: monmap: 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:53.231 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 librados: starting msgr at 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 librados: starting objecter 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 Processor -- start 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 -- start start 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 librados: setting wanted keys 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 1 librados: calling monclient init 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.237+0000 7f1250df6640 10 monclient: init 2024-03-29T11:15:53.233 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-b con 0x7f124c0b39f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-a con 0x7f124c0ae080 addr v1:172.21.15.105:6789/0 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient(hunting): picked mon.noname-c con 0x7f124c142040 addr v1:172.21.15.105:6790/0 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0ac970 con 0x7f124c0ae080 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0acc50 con 0x7f124c0b39f0 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0aa5f0 con 0x7f124c142040 2024-03-29T11:15:53.234 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f1250df6640 10 monclient(hunting): authenticate will time out at 1227.514248s 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f124ad9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f124c142040 0x7f124c1e1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:51138/0 (socket says 172.21.15.105:51138) 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f124ad9d640 1 -- 172.21.15.105:0/2928531570 learned_addr learned my addr 172.21.15.105:0/2928531570 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f123affd640 10 client.?.objecter ms_handle_connect 0x7f124c142040 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f123affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f123affd640 10 client.?.objecter ms_handle_connect 0x7f124c0b39f0 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.238+0000 7f123affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:53.235 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 client.?.objecter ms_handle_connect 0x7f124c0ae080 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442403038 0 0) 0x7f124c0acc50 con 0x7f124c0b39f0 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c004eb0 con 0x7f124c0b39f0 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480428398 0 0) 0x7f124c0ac970 con 0x7f124c0ae080 2024-03-29T11:15:53.236 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0051d0 con 0x7f124c0ae080 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107366714 0 0) 0x7f124c0aa5f0 con 0x7f124c142040 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:53.237 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c0ac970 con 0x7f124c142040 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221080285 0 0) 0x7f122c0051d0 con 0x7f124c0ae080 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f124c0aa5f0 con 0x7f124c0ae080 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f12400018b0 con 0x7f124c0ae080 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:53.238 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): dump: 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:53.239 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3520113640 0 0) 0x7f124c0ac970 con 0x7f124c142040 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f122c0051d0 con 0x7f124c142040 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1244001a10 con 0x7f124c142040 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.239+0000 7f123affd640 10 monclient(hunting): dump: 2024-03-29T11:15:53.240 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:53.241 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710468439 0 0) 0x7f124c0aa5f0 con 0x7f124c0ae080 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 >> v1:172.21.15.105:6790/0 conn(0x7f124c142040 legacy=0x7f124c1e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 >> v1:172.21.15.110:6789/0 conn(0x7f124c0b39f0 legacy=0x7f124c0ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 monclient: found mon.a 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.242 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c0abaa0 con 0x7f124c0ae080 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: _finish_auth 0 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 10 monclient: _renew_subs 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c1e35f0 con 0x7f124c0ae080 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:53.243 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 10 monclient: _renew_subs 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1e3960 con 0x7f124c0ae080 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f1250df6640 1 librados: init done 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1240001a90 con 0x7f124c0ae080 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f12400038c0 con 0x7f124c0ae080 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:53.244 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123affd640 10 monclient: dump: 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:15:53.245 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.240+0000 7f123bfff640 4 set_mon_vals no callback set 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.241+0000 7f1250df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.241+0000 7f1250df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.241+0000 7f1250df6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.241+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1230005080 con 0x7f124c0ae080 2024-03-29T11:15:53.246 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.242+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f1240019890 con 0x7f124c0ae080 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f1240083820 con 0x7f124c0ae080 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 10 client.4488.objecter ms_dispatch 0x7f124c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f124a59c640 10 monclient: get_auth_request con 0x7f122c06e120 auth_method 0 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:15:53.247 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.243+0000 7f123affd640 10 client.4488.objecter ms_handle_connect 0x7f122c06e120 2024-03-29T11:15:53.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.250+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f124004f100 con 0x7f124c0ae080 2024-03-29T11:15:53.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.251+0000 7f123affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:53.248 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.251+0000 7f123affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:53.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f1250df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f1250df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f1250df6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:15:53.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1230005330 con 0x7f124c0ae080 2024-03-29T11:15:53.566 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f123affd640 1 -- 172.21.15.105:0/2928531570 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f1240051030 con 0x7f124c0ae080 2024-03-29T11:15:53.566 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f123affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.566 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.587+0000 7f123affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:53.567 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:15:53.567 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:53.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 10 librados: watch_flush enter 2024-03-29T11:15:53.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 10 librados: watch_flush exit 2024-03-29T11:15:53.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:15:53.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:15:53.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 >> v1:172.21.15.110:6816/1087970004 conn(0x7f122c06e120 legacy=0x7f122c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 10 monclient: shutdown 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 >> v1:172.21.15.105:6789/0 conn(0x7f124c0ae080 legacy=0x7f124c141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 shutdown_connections 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 >> 172.21.15.105:0/2928531570 conn(0x7f124c066a00 msgr2=0x7f124c13f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 shutdown_connections 2024-03-29T11:15:53.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 -- 172.21.15.105:0/2928531570 wait complete. 2024-03-29T11:15:53.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:15:53.591+0000 7f1250df6640 1 librados: shutdown 2024-03-29T11:15:53.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:00.391 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:00.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.574+0000 7f11a1463640 10 monclient: get_monmap_and_config 2024-03-29T11:16:00.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.574+0000 7f11a1463640 10 monclient: build_initial_monmap 2024-03-29T11:16:00.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.574+0000 7f11a1463640 10 monclient: monmap: 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:00.575951+0000 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:00.575951+0000 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:00.553 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.576+0000 7f11a1463640 1 Processor -- start 2024-03-29T11:16:00.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.576+0000 7f11a1463640 1 -- start start 2024-03-29T11:16:00.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.576+0000 7f11a1463640 10 monclient: init 2024-03-29T11:16:00.554 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.577+0000 7f11a1463640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:00.555 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.577+0000 7f11a1463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:00.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.577+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-b con 0x7f119c0b0770 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.581+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-c con 0x7f119c14a700 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.581+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-a con 0x7f119c0ac5a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.581+0000 7f11a1463640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c0aba70 con 0x7f119c0ac5a0 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f11a1463640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c0aecb0 con 0x7f119c14a700 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f11a1463640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c0af150 con 0x7f119c0b0770 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f11a1463640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f11a1463640 10 monclient(hunting): authenticate will time out at 1234.858314s 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f119affd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f119c0b0770 0x7f119c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60875/0 (socket says 172.21.15.105:60875) 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f119affd640 1 -- 172.21.15.105:0/3658765531 learned_addr learned my addr 172.21.15.105:0/3658765531 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:00.560 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.582+0000 7f119a7fc640 1 --1- 172.21.15.105:0/3658765531 >> v1:172.21.15.110:6789/0 conn(0x7f119c14a700 0x7f119c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:51765/0 (socket says 172.21.15.105:51765) 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531230224 0 0) 0x7f119c0aecb0 con 0x7f119c14a700 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:16:00.563 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1184003520 con 0x7f119c14a700 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288662260 0 0) 0x7f119c0af150 con 0x7f119c0b0770 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.583+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f119c0aecb0 con 0x7f119c0b0770 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613663835 0 0) 0x7f119c0aecb0 con 0x7f119c0b0770 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 >> v1:172.21.15.110:6789/0 conn(0x7f119c14a700 legacy=0x7f119c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 >> v1:172.21.15.105:6789/0 conn(0x7f119c0ac5a0 legacy=0x7f119c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:00.564 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 1 monclient: found mon.noname-b 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.584+0000 7f1199ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c0ac9d0 con 0x7f119c0b0770 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f11940019f0 con 0x7f119c0b0770 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f1199ffb640 10 monclient: dump: 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.565 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f11a1463640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:16:00.566 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.585+0000 7f11a1463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1194001c10 con 0x7f119c0b0770 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 1 -- 172.21.15.105:0/3658765531 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1194001fa0 con 0x7f119c0b0770 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.588+0000 7f1199ffb640 10 monclient: dump: 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:00.567 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.589+0000 7f11a1463640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.589+0000 7f11a1463640 4 set_mon_vals no callback set 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 10 monclient: shutdown 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 1 -- 172.21.15.105:0/3658765531 >> v1:172.21.15.105:6790/0 conn(0x7f119c0b0770 legacy=0x7f119c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.568 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 1 -- 172.21.15.105:0/3658765531 shutdown_connections 2024-03-29T11:16:00.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 1 -- 172.21.15.105:0/3658765531 >> 172.21.15.105:0/3658765531 conn(0x7f119c06b660 msgr2=0x7f119c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:00.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 1 -- 172.21.15.105:0/3658765531 shutdown_connections 2024-03-29T11:16:00.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.590+0000 7f11a1463640 1 -- 172.21.15.105:0/3658765531 wait complete. 2024-03-29T11:16:00.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.591+0000 7f11a1463640 10 monclient: build_initial_monmap 2024-03-29T11:16:00.569 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.591+0000 7f11a1463640 10 monclient: monmap: 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.592+0000 7f11a1463640 1 librados: starting msgr at 2024-03-29T11:16:00.570 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.592+0000 7f11a1463640 1 librados: starting objecter 2024-03-29T11:16:00.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.592+0000 7f11a1463640 1 Processor -- start 2024-03-29T11:16:00.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.593+0000 7f11a1463640 1 -- start start 2024-03-29T11:16:00.571 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.593+0000 7f11a1463640 1 librados: setting wanted keys 2024-03-29T11:16:00.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.593+0000 7f11a1463640 1 librados: calling monclient init 2024-03-29T11:16:00.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.593+0000 7f11a1463640 10 monclient: init 2024-03-29T11:16:00.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.594+0000 7f11a1463640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:00.572 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.594+0000 7f11a1463640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-c con 0x7f119c0ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-a con 0x7f119c14a700 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 10 monclient(hunting): picked mon.noname-b con 0x7f119c1df5c0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c1239d0 con 0x7f119c14a700 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c126260 con 0x7f119c1df5c0 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c145b30 con 0x7f119c0ac5a0 2024-03-29T11:16:00.574 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:00.575 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.596+0000 7f11a1463640 10 monclient(hunting): authenticate will time out at 1234.872561s 2024-03-29T11:16:00.578 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.601+0000 7f119affd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f119c0ac5a0 0x7f119c144e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60981/0 (socket says 172.21.15.105:60981) 2024-03-29T11:16:00.578 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.601+0000 7f119affd640 1 -- 172.21.15.105:0/1857391985 learned_addr learned my addr 172.21.15.105:0/1857391985 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:00.578 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.601+0000 7f119a7fc640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f119c14a700 0x7f119c126780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46761/0 (socket says 172.21.15.105:46761) 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 client.?.objecter ms_handle_connect 0x7f119c0ac5a0 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334780587 0 0) 0x7f119c145b30 con 0x7f119c0ac5a0 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:16:00.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1188004eb0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 client.?.objecter ms_handle_connect 0x7f119c14a700 2024-03-29T11:16:00.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.602+0000 7f11837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1631567699 0 0) 0x7f1188004eb0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f119c145b30 con 0x7f119c0ac5a0 2024-03-29T11:16:00.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f11940019f0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): dump: 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.591 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:00.592 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097864611 0 0) 0x7f119c1239d0 con 0x7f119c14a700 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1188004eb0 con 0x7f119c14a700 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4022177519 0 0) 0x7f119c145b30 con 0x7f119c0ac5a0 2024-03-29T11:16:00.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.603+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 >> v1:172.21.15.110:6789/0 conn(0x7f119c1df5c0 legacy=0x7f119c1e6cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 >> v1:172.21.15.105:6789/0 conn(0x7f119c14a700 legacy=0x7f119c126780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 1 monclient: found mon.c 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c125cb0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 10 monclient: _renew_subs 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c125f00 con 0x7f119c0ac5a0 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 10 monclient: _renew_subs 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f119c145730 con 0x7f119c0ac5a0 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.604+0000 7f11a1463640 1 librados: init done 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1194004e10 con 0x7f119c0ac5a0 2024-03-29T11:16:00.596 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1194005220 con 0x7f119c0ac5a0 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11837fe640 10 monclient: dump: 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:00.597 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:00.598 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.605+0000 7f11997fa640 4 set_mon_vals no callback set 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.608+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f1194005380 con 0x7f119c0ac5a0 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f1194084fe0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f11837fe640 10 client.4565.objecter ms_dispatch 0x7f119c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:00.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f119a7fc640 10 monclient: get_auth_request con 0x7f118806dbc0 auth_method 0 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f11837fe640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f11837fe640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.610+0000 7f11837fe640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.611+0000 7f11837fe640 10 client.4565.objecter ms_handle_connect 0x7f118806dbc0 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.611+0000 7f11a1463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.611+0000 7f11a1463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.612+0000 7f11a1463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.612+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c005080 con 0x7f119c0ac5a0 2024-03-29T11:16:00.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.620+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11940508c0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.620+0000 7f11837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.602 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.620+0000 7f11837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:00.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.989+0000 7f11a1463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.989+0000 7f11a1463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.989+0000 7f11a1463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:00.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.989+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f117c005330 con 0x7f119c0ac5a0 2024-03-29T11:16:00.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.990+0000 7f11837fe640 1 -- 172.21.15.105:0/1857391985 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f11940527f0 con 0x7f119c0ac5a0 2024-03-29T11:16:00.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.990+0000 7f11837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.990+0000 7f11837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:00.969 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:00.969 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:00.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 10 librados: watch_flush enter 2024-03-29T11:16:00.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 10 librados: watch_flush exit 2024-03-29T11:16:00.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:16:00.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 >> v1:172.21.15.110:6816/1087970004 conn(0x7f118806dbc0 legacy=0x7f1188070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 10 monclient: shutdown 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.994+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 >> v1:172.21.15.105:6790/0 conn(0x7f119c0ac5a0 legacy=0x7f119c144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.995+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 shutdown_connections 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.995+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 >> 172.21.15.105:0/1857391985 conn(0x7f119c06b660 msgr2=0x7f119c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.995+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 shutdown_connections 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.995+0000 7f11a1463640 1 -- 172.21.15.105:0/1857391985 wait complete. 2024-03-29T11:16:00.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:00.995+0000 7f11a1463640 1 librados: shutdown 2024-03-29T11:16:00.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:07.796 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:07.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.977+0000 7fe752596640 10 monclient: get_monmap_and_config 2024-03-29T11:16:07.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.977+0000 7fe752596640 10 monclient: build_initial_monmap 2024-03-29T11:16:07.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.977+0000 7fe752596640 10 monclient: monmap: 2024-03-29T11:16:07.956 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:07.978706+0000 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:07.978706+0000 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.978+0000 7fe752596640 1 Processor -- start 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.978+0000 7fe752596640 1 -- start start 2024-03-29T11:16:07.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.978+0000 7fe752596640 10 monclient: init 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c0ae250 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c13f970 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c142800 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:07.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c0afe00 con 0x7fe74c142800 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c142c10 con 0x7fe74c0ae250 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c0afb20 con 0x7fe74c13f970 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.979+0000 7fe752596640 10 monclient(hunting): authenticate will time out at 1242.255961s 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74bfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe74c0ae250 0x7fe74c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:41775/0 (socket says 172.21.15.105:41775) 2024-03-29T11:16:07.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74bfff640 1 -- 172.21.15.105:0/3894123266 learned_addr learned my addr 172.21.15.105:0/3894123266 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888627036 0 0) 0x7fe74c142c10 con 0x7fe74c0ae250 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.983+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe730003520 con 0x7fe74c0ae250 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 800609257 0 0) 0x7fe730003520 con 0x7fe74c0ae250 2024-03-29T11:16:07.964 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 >> v1:172.21.15.105:6790/0 conn(0x7fe74c13f970 legacy=0x7fe74c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 >> v1:172.21.15.105:6789/0 conn(0x7fe74c142800 legacy=0x7fe74c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 monclient: found mon.noname-c 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c0aab80 con 0x7fe74c0ae250 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: _finish_auth 0 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe752596640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe752596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe73c001690 con 0x7fe74c0ae250 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:07.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe74affd640 10 monclient: dump: 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.984+0000 7fe752596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.986+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe73c001920 con 0x7fe74c0ae250 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.986+0000 7fe74affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.986+0000 7fe752596640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:07.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.986+0000 7fe752596640 4 set_mon_vals no callback set 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe74affd640 1 -- 172.21.15.105:0/3894123266 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe73c001cb0 con 0x7fe74c0ae250 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe74affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe74affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe74affd640 10 monclient: dump: 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:07.967 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe752596640 10 monclient: shutdown 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe752596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.987+0000 7fe752596640 1 -- 172.21.15.105:0/3894123266 >> v1:172.21.15.110:6789/0 conn(0x7fe74c0ae250 legacy=0x7fe74c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.988+0000 7fe752596640 1 -- 172.21.15.105:0/3894123266 shutdown_connections 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.988+0000 7fe752596640 1 -- 172.21.15.105:0/3894123266 >> 172.21.15.105:0/3894123266 conn(0x7fe74c0b35e0 msgr2=0x7fe74c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.988+0000 7fe752596640 1 -- 172.21.15.105:0/3894123266 shutdown_connections 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.988+0000 7fe752596640 1 -- 172.21.15.105:0/3894123266 wait complete. 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.989+0000 7fe752596640 10 monclient: build_initial_monmap 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.989+0000 7fe752596640 10 monclient: monmap: 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:07.969 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.989+0000 7fe752596640 1 librados: starting msgr at 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.989+0000 7fe752596640 1 librados: starting objecter 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.990+0000 7fe752596640 1 Processor -- start 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.990+0000 7fe752596640 1 -- start start 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.990+0000 7fe752596640 1 librados: setting wanted keys 2024-03-29T11:16:07.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.990+0000 7fe752596640 1 librados: calling monclient init 2024-03-29T11:16:07.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.990+0000 7fe752596640 10 monclient: init 2024-03-29T11:16:07.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:07.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c0ae250 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c13f970 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c142800 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c1386d0 con 0x7fe74c0ae250 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c1459d0 con 0x7fe74c13f970 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c145cb0 con 0x7fe74c142800 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.991+0000 7fe752596640 10 monclient(hunting): authenticate will time out at 1242.267743s 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe74bfff640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fe74c0ae250 0x7fe74c1dc8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:50215/0 (socket says 172.21.15.105:50215) 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe74bfff640 1 -- 172.21.15.105:0/2208176910 learned_addr learned my addr 172.21.15.105:0/2208176910 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 client.?.objecter ms_handle_connect 0x7fe74c0ae250 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:07.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514058316 0 0) 0x7fe74c1386d0 con 0x7fe74c0ae250 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.995+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe734004eb0 con 0x7fe74c0ae250 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 client.?.objecter ms_handle_connect 0x7fe74c13f970 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323792366 0 0) 0x7fe74c1459d0 con 0x7fe74c13f970 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.996+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c1386d0 con 0x7fe74c13f970 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 131619790 0 0) 0x7fe74c1386d0 con 0x7fe74c13f970 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe74c1459d0 con 0x7fe74c13f970 2024-03-29T11:16:07.977 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe738001930 con 0x7fe74c13f970 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574434884 0 0) 0x7fe74c1459d0 con 0x7fe74c13f970 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 >> v1:172.21.15.105:6790/0 conn(0x7fe74c142800 legacy=0x7fe74c1e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 >> v1:172.21.15.105:6789/0 conn(0x7fe74c0ae250 legacy=0x7fe74c1dc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 monclient: found mon.b 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c145f90 con 0x7fe74c13f970 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe748ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe752596640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe752596640 10 monclient: _renew_subs 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe752596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:07.979 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.997+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c144e10 con 0x7fe74c13f970 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe738003610 con 0x7fe74c13f970 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe752596640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe752596640 10 monclient: _renew_subs 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe752596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:07.980 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe738003920 con 0x7fe74c13f970 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe74a7fc640 4 set_mon_vals no callback set 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c13b780 con 0x7fe74c13f970 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe752596640 1 librados: init done 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.998+0000 7fe748ff9640 10 monclient: dump: 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:07.981 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.999+0000 7fe752596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.999+0000 7fe752596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.999+0000 7fe752596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:07.999+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c13ba80 con 0x7fe74c13f970 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fe738003a80 con 0x7fe74c13f970 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fe738083830 con 0x7fe74c13f970 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 10 client.4495.objecter ms_dispatch 0x7fe74c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe74bfff640 10 monclient: get_auth_request con 0x7fe73406dd30 auth_method 0 2024-03-29T11:16:07.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.001+0000 7fe748ff9640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:16:07.983 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.002+0000 7fe748ff9640 10 client.4495.objecter ms_handle_connect 0x7fe73406dd30 2024-03-29T11:16:07.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.007+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe73804f110 con 0x7fe74c13f970 2024-03-29T11:16:07.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.007+0000 7fe748ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.984 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.007+0000 7fe748ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:08.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.366+0000 7fe752596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.366+0000 7fe752596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.367+0000 7fe752596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:08.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.367+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe74c075f10 con 0x7fe74c13f970 2024-03-29T11:16:08.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.368+0000 7fe748ff9640 1 -- 172.21.15.105:0/2208176910 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fe738051040 con 0x7fe74c13f970 2024-03-29T11:16:08.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.368+0000 7fe748ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:08.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.368+0000 7fe748ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:08.347 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:08.347 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:08.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.374+0000 7fe752596640 10 librados: watch_flush enter 2024-03-29T11:16:08.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.374+0000 7fe752596640 10 librados: watch_flush exit 2024-03-29T11:16:08.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.374+0000 7fe752596640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:16:08.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.374+0000 7fe752596640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:16:08.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.375+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 >> v1:172.21.15.110:6816/1087970004 conn(0x7fe73406dd30 legacy=0x7fe7340701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.375+0000 7fe752596640 10 monclient: shutdown 2024-03-29T11:16:08.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.375+0000 7fe752596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:08.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.375+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 >> v1:172.21.15.110:6789/0 conn(0x7fe74c13f970 legacy=0x7fe74c144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.377+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 shutdown_connections 2024-03-29T11:16:08.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.377+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 >> 172.21.15.105:0/2208176910 conn(0x7fe74c0b35e0 msgr2=0x7fe74c142000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:08.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.377+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 shutdown_connections 2024-03-29T11:16:08.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.377+0000 7fe752596640 1 -- 172.21.15.105:0/2208176910 wait complete. 2024-03-29T11:16:08.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:08.377+0000 7fe752596640 1 librados: shutdown 2024-03-29T11:16:08.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:15.180 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:15.328 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.351+0000 7f8cc78c1640 10 monclient: get_monmap_and_config 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.351+0000 7f8cc78c1640 10 monclient: build_initial_monmap 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.351+0000 7f8cc78c1640 10 monclient: monmap: 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:15.352449+0000 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:15.352449+0000 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:15.329 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.352+0000 7f8cc78c1640 1 Processor -- start 2024-03-29T11:16:15.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.352+0000 7f8cc78c1640 1 -- start start 2024-03-29T11:16:15.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.352+0000 7f8cc78c1640 10 monclient: init 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc01457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc013f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc00a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc00aaef0 con 0x7f8cc01457f0 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0145bd0 con 0x7f8cc013f650 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0145eb0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc78c1640 10 monclient(hunting): authenticate will time out at 1249.629354s 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc4e5c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8cc013f650 0x7f8cc00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:58776/0 (socket says 172.21.15.105:58776) 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.353+0000 7f8cc4e5c640 1 -- 172.21.15.105:0/1529978119 learned_addr learned my addr 172.21.15.105:0/1529978119 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:15.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949303495 0 0) 0x7f8cc0145bd0 con 0x7f8cc013f650 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac003520 con 0x7f8cc013f650 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428036252 0 0) 0x7f8cc00aaef0 con 0x7f8cc01457f0 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0145bd0 con 0x7f8cc01457f0 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2013161340 0 0) 0x7f8cac003520 con 0x7f8cc013f650 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 >> v1:172.21.15.105:6790/0 conn(0x7f8cc00a9d70 legacy=0x7f8cc014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 >> v1:172.21.15.105:6789/0 conn(0x7f8cc01457f0 legacy=0x7f8cc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.336 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 monclient: found mon.noname-c 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0146340 con 0x7f8cc013f650 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8cb8001990 con 0x7f8cc013f650 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cb7fff640 10 monclient: dump: 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.337 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cc78c1640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.354+0000 7f8cc78c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb8001af0 con 0x7f8cc013f650 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 1 -- 172.21.15.105:0/1529978119 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8cb8001f40 con 0x7f8cc013f650 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.338 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cb7fff640 10 monclient: dump: 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 4 set_mon_vals no callback set 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 10 monclient: shutdown 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:15.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 1 -- 172.21.15.105:0/1529978119 >> v1:172.21.15.110:6789/0 conn(0x7f8cc013f650 legacy=0x7f8cc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 1 -- 172.21.15.105:0/1529978119 shutdown_connections 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.355+0000 7f8cc78c1640 1 -- 172.21.15.105:0/1529978119 >> 172.21.15.105:0/1529978119 conn(0x7f8cc0066a00 msgr2=0x7f8cc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.356+0000 7f8cc78c1640 1 -- 172.21.15.105:0/1529978119 shutdown_connections 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.356+0000 7f8cc78c1640 1 -- 172.21.15.105:0/1529978119 wait complete. 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.356+0000 7f8cc78c1640 10 monclient: build_initial_monmap 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.356+0000 7f8cc78c1640 10 monclient: monmap: 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:15.340 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 librados: starting msgr at 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 librados: starting objecter 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 Processor -- start 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 -- start start 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 librados: setting wanted keys 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 librados: calling monclient init 2024-03-29T11:16:15.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient: init 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc00a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc013f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc01457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc00ab9e0 con 0x7f8cc01457f0 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc00ac860 con 0x7f8cc00a9d70 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0137c30 con 0x7f8cc013f650 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:15.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.357+0000 7f8cc78c1640 10 monclient(hunting): authenticate will time out at 1249.633962s 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cc565d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8cc00a9d70 0x7f8cc01d74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:59080/0 (socket says 172.21.15.105:59080) 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cc565d640 1 -- 172.21.15.105:0/2200556464 learned_addr learned my addr 172.21.15.105:0/2200556464 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f8cc00a9d70 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f8cc013f650 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978857644 0 0) 0x7f8cc00ac860 con 0x7f8cc00a9d70 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f8cc01457f0 2024-03-29T11:16:15.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764855725 0 0) 0x7f8cc0137c30 con 0x7f8cc013f650 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.358+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc00ac860 con 0x7f8cc013f650 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654622297 0 0) 0x7f8ca4004eb0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc0137c30 con 0x7f8cc00a9d70 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8cb0001970 con 0x7f8cc00a9d70 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:15.348 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1911535379 0 0) 0x7f8cc00ac860 con 0x7f8cc013f650 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cc013f650 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8cb8001e00 con 0x7f8cc013f650 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:15.349 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247285151 0 0) 0x7f8cc00ab9e0 con 0x7f8cc01457f0 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:15.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.359+0000 7f8cb5ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc00ac860 con 0x7f8cc01457f0 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3697623810 0 0) 0x7f8cc0137c30 con 0x7f8cc00a9d70 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 >> v1:172.21.15.105:6790/0 conn(0x7f8cc013f650 legacy=0x7f8cc0136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 >> v1:172.21.15.105:6789/0 conn(0x7f8cc01457f0 legacy=0x7f8cc01e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 monclient: found mon.b 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0141eb0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:16:15.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 10 monclient: _renew_subs 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc0142a50 con 0x7f8cc00a9d70 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 10 monclient: _renew_subs 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc0145040 con 0x7f8cc00a9d70 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cc78c1640 1 librados: init done 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb0001ad0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8cb00038c0 con 0x7f8cc00a9d70 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb5ffb640 10 monclient: dump: 2024-03-29T11:16:15.352 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.360+0000 7f8cb77fe640 4 set_mon_vals no callback set 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8cb0003a20 con 0x7f8cc00a9d70 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cc78c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cc78c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cc78c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca0005080 con 0x7f8cc00a9d70 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8cb0084a00 con 0x7f8cc00a9d70 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 10 client.4501.objecter ms_dispatch 0x7f8cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cb5ffb640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.363+0000 7f8cc4e5c640 10 monclient: get_auth_request con 0x7f8ca406e150 auth_method 0 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.364+0000 7f8cb5ffb640 10 client.4501.objecter ms_handle_connect 0x7f8ca406e150 2024-03-29T11:16:15.354 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.372+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cb0050110 con 0x7f8cc00a9d70 2024-03-29T11:16:15.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.372+0000 7f8cb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:15.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.372+0000 7f8cb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:15.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.727+0000 7f8cc78c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.727+0000 7f8cc78c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.727+0000 7f8cc78c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:15.705 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.727+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ca0005330 con 0x7f8cc00a9d70 2024-03-29T11:16:15.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.728+0000 7f8cb5ffb640 1 -- 172.21.15.105:0/2200556464 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8cb0052210 con 0x7f8cc00a9d70 2024-03-29T11:16:15.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.728+0000 7f8cb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.706 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.728+0000 7f8cb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:15.706 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:15.706 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 10 librados: watch_flush enter 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 10 librados: watch_flush exit 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 >> v1:172.21.15.110:6816/1087970004 conn(0x7f8ca406e150 legacy=0x7f8ca4070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 10 monclient: shutdown 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:15.710 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 >> v1:172.21.15.110:6789/0 conn(0x7f8cc00a9d70 legacy=0x7f8cc01d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 shutdown_connections 2024-03-29T11:16:15.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 >> 172.21.15.105:0/2200556464 conn(0x7f8cc0066a00 msgr2=0x7f8cc00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:15.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.732+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 shutdown_connections 2024-03-29T11:16:15.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.733+0000 7f8cc78c1640 1 -- 172.21.15.105:0/2200556464 wait complete. 2024-03-29T11:16:15.711 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:15.733+0000 7f8cc78c1640 1 librados: shutdown 2024-03-29T11:16:15.725 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:22.530 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.701+0000 7ff411281640 10 monclient: get_monmap_and_config 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.701+0000 7ff411281640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.701+0000 7ff411281640 10 monclient: monmap: 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:22.702592+0000 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:22.702592+0000 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:22.679 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.681 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.703+0000 7ff411281640 1 Processor -- start 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.703+0000 7ff411281640 1 -- start start 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.703+0000 7ff411281640 10 monclient: init 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient(hunting): picked mon.noname-c con 0x7ff404038a00 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient(hunting): picked mon.noname-b con 0x7ff404038de0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient(hunting): picked mon.noname-a con 0x7ff404030680 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040c54e0 con 0x7ff404030680 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff404036450 con 0x7ff404038a00 2024-03-29T11:16:22.682 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff404039860 con 0x7ff404038de0 2024-03-29T11:16:22.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff411281640 10 monclient(hunting): authenticate will time out at 1256.980132s 2024-03-29T11:16:22.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff40a7fc640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7ff404038de0 0x7ff4040cadf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:39732/0 (socket says 172.21.15.105:39732) 2024-03-29T11:16:22.683 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.704+0000 7ff40a7fc640 1 -- 172.21.15.105:0/842278864 learned_addr learned my addr 172.21.15.105:0/842278864 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:22.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.706+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878710359 0 0) 0x7ff404039860 con 0x7ff404038de0 2024-03-29T11:16:22.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.684 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4003520 con 0x7ff404038de0 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175666126 0 0) 0x7ff4040c54e0 con 0x7ff404030680 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff404039860 con 0x7ff404030680 2024-03-29T11:16:22.685 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093089065 0 0) 0x7ff3f4003520 con 0x7ff404038de0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 >> v1:172.21.15.110:6789/0 conn(0x7ff404038a00 legacy=0x7ff404033540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.707+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 >> v1:172.21.15.105:6789/0 conn(0x7ff404030680 legacy=0x7ff4040d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 1 monclient: found mon.noname-b 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040356b0 con 0x7ff404038de0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3fc001690 con 0x7ff404038de0 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:22.686 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: dump: 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff411281640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:16:22.687 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff411281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3fc001860 con 0x7ff404038de0 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 1 -- 172.21.15.105:0/842278864 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3fc001c70 con 0x7ff404038de0 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:22.688 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.708+0000 7ff409ffb640 10 monclient: dump: 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.709+0000 7ff411281640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.709+0000 7ff411281640 4 set_mon_vals no callback set 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 10 monclient: shutdown 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 1 -- 172.21.15.105:0/842278864 >> v1:172.21.15.105:6790/0 conn(0x7ff404038de0 legacy=0x7ff4040cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 1 -- 172.21.15.105:0/842278864 shutdown_connections 2024-03-29T11:16:22.689 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 1 -- 172.21.15.105:0/842278864 >> 172.21.15.105:0/842278864 conn(0x7ff404005c20 msgr2=0x7ff404006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.710+0000 7ff411281640 1 -- 172.21.15.105:0/842278864 shutdown_connections 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.711+0000 7ff411281640 1 -- 172.21.15.105:0/842278864 wait complete. 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.711+0000 7ff411281640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.711+0000 7ff411281640 10 monclient: monmap: 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:22.690 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 librados: starting msgr at 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 librados: starting objecter 2024-03-29T11:16:22.691 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 Processor -- start 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 -- start start 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 librados: setting wanted keys 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 1 librados: calling monclient init 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.712+0000 7ff411281640 10 monclient: init 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient(hunting): picked mon.noname-b con 0x7ff404030680 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient(hunting): picked mon.noname-c con 0x7ff404038a00 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient(hunting): picked mon.noname-a con 0x7ff4040c8290 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040301d0 con 0x7ff4040c8290 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40402f300 con 0x7ff404030680 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40402e920 con 0x7ff404038a00 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.713+0000 7ff411281640 10 monclient(hunting): authenticate will time out at 1256.989896s 2024-03-29T11:16:22.692 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.714+0000 7ff40a7fc640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7ff404038a00 0x7ff4040bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:42230/0 (socket says 172.21.15.105:42230) 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.714+0000 7ff40a7fc640 1 -- 172.21.15.105:0/309174537 learned_addr learned my addr 172.21.15.105:0/309174537 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.714+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff404038a00 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.714+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740739181 0 0) 0x7ff40402e920 con 0x7ff404038a00 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:16:22.693 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.715+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8004eb0 con 0x7ff404038a00 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518451029 0 0) 0x7ff3f8004eb0 con 0x7ff404038a00 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f8005300 con 0x7ff404038a00 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff404030680 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3fc001b70 con 0x7ff404038a00 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 monclient(hunting): dump: 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.696 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.718+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782906296 0 0) 0x7ff40402f300 con 0x7ff404030680 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8004eb0 con 0x7ff404030680 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 client.?.objecter ms_handle_connect 0x7ff4040c8290 2024-03-29T11:16:22.697 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3898302173 0 0) 0x7ff3f8005300 con 0x7ff404038a00 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 >> v1:172.21.15.110:6789/0 conn(0x7ff404030680 legacy=0x7ff40402fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 >> v1:172.21.15.105:6789/0 conn(0x7ff4040c8290 legacy=0x7ff4040c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 monclient: found mon.c 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40402ec00 con 0x7ff404038a00 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:16:22.698 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 10 monclient: _renew_subs 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40402eee0 con 0x7ff404038a00 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 10 monclient: _renew_subs 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40402d170 con 0x7ff404038a00 2024-03-29T11:16:22.699 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.719+0000 7ff411281640 1 librados: init done 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3fc004a90 con 0x7ff404038a00 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7ff3fc004da0 con 0x7ff404038a00 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff3f37fe640 10 monclient: dump: 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:22.700 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.720+0000 7ff4097fa640 4 set_mon_vals no callback set 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.722+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7ff3fc01ad70 con 0x7ff404038a00 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff40affd640 10 monclient: get_auth_request con 0x7ff3f806df60 auth_method 0 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7ff3fc0547e0 con 0x7ff404038a00 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff3f37fe640 10 client.4580.objecter ms_dispatch 0x7ff40c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:22.701 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff3f37fe640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff3f37fe640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff3f37fe640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff411281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff411281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff411281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.723+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec005080 con 0x7ff404038a00 2024-03-29T11:16:22.702 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.725+0000 7ff3f37fe640 10 client.4580.objecter ms_handle_connect 0x7ff3f806df60 2024-03-29T11:16:22.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.732+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3fc050a80 con 0x7ff404038a00 2024-03-29T11:16:22.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.732+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.709 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:22.732+0000 7ff3f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:23.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.064+0000 7ff411281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.064+0000 7ff411281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.064+0000 7ff411281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:23.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.064+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3ec005330 con 0x7ff404038a00 2024-03-29T11:16:23.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.065+0000 7ff3f37fe640 1 -- 172.21.15.105:0/309174537 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7ff3fc050c20 con 0x7ff404038a00 2024-03-29T11:16:23.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.065+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.065+0000 7ff3f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:23.043 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:23.043 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:23.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 10 librados: watch_flush enter 2024-03-29T11:16:23.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 10 librados: watch_flush exit 2024-03-29T11:16:23.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 >> v1:172.21.15.110:6816/1087970004 conn(0x7ff3f806df60 legacy=0x7ff3f8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 10 monclient: shutdown 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 >> v1:172.21.15.105:6790/0 conn(0x7ff404038a00 legacy=0x7ff4040bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 shutdown_connections 2024-03-29T11:16:23.047 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.069+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 >> 172.21.15.105:0/309174537 conn(0x7ff404005c20 msgr2=0x7ff4040340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:23.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.070+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 shutdown_connections 2024-03-29T11:16:23.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.070+0000 7ff411281640 1 -- 172.21.15.105:0/309174537 wait complete. 2024-03-29T11:16:23.048 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:23.071+0000 7ff411281640 1 librados: shutdown 2024-03-29T11:16:23.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:29.874 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:30.021 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.043+0000 7fadd7d38640 10 monclient: get_monmap_and_config 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.043+0000 7fadd7d38640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.043+0000 7fadd7d38640 10 monclient: monmap: 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:30.044432+0000 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:30.044432+0000 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:30.022 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:30.023 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:30.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.044+0000 7fadd7d38640 1 Processor -- start 2024-03-29T11:16:30.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.044+0000 7fadd7d38640 1 -- start start 2024-03-29T11:16:30.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.044+0000 7fadd7d38640 10 monclient: init 2024-03-29T11:16:30.023 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.045+0000 7fadd7d38640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.045+0000 7fadd7d38640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.045+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-c con 0x7fadd00b4100 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.045+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-a con 0x7fadd013ee90 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.045+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-b con 0x7fadd00af900 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.046+0000 7fadd7d38640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00afd10 con 0x7fadd013ee90 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.046+0000 7fadd7d38640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00af6f0 con 0x7fadd00b4100 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.046+0000 7fadd7d38640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00aaf60 con 0x7fadd00af900 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.046+0000 7fadd7d38640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.024 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.046+0000 7fadd7d38640 10 monclient(hunting): authenticate will time out at 1264.322014s 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.049+0000 7fadd52d3640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fadd013ee90 0x7fadd00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:34479/0 (socket says 172.21.15.105:34479) 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.049+0000 7fadd52d3640 1 -- 172.21.15.105:0/4201330332 learned_addr learned my addr 172.21.15.105:0/4201330332 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119158660 0 0) 0x7fadd00aaf60 con 0x7fadd00af900 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.027 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc003520 con 0x7fadd00af900 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840845957 0 0) 0x7fadd00afd10 con 0x7fadd013ee90 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd00aaf60 con 0x7fadd013ee90 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727491919 0 0) 0x7fadbc003520 con 0x7fadd00af900 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 >> v1:172.21.15.110:6789/0 conn(0x7fadd00b4100 legacy=0x7fadd00a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:30.028 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 >> v1:172.21.15.105:6789/0 conn(0x7fadd013ee90 legacy=0x7fadd00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 monclient: found mon.noname-b 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd013e970 con 0x7fadd00af900 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd7d38640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd7d38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fadcc0015c0 con 0x7fadd00af900 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:30.029 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: dump: 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd7d38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 1 -- 172.21.15.105:0/4201330332 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadcc0017e0 con 0x7fadd00af900 2024-03-29T11:16:30.030 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd4ad2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd7d38640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.050+0000 7fadd7d38640 4 set_mon_vals no callback set 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.051+0000 7fadd7d38640 10 monclient: shutdown 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.051+0000 7fadd7d38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.051+0000 7fadd7d38640 1 -- 172.21.15.105:0/4201330332 >> v1:172.21.15.105:6790/0 conn(0x7fadd00af900 legacy=0x7fadd013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.052+0000 7fadd7d38640 1 -- 172.21.15.105:0/4201330332 shutdown_connections 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.052+0000 7fadd7d38640 1 -- 172.21.15.105:0/4201330332 >> 172.21.15.105:0/4201330332 conn(0x7fadd00b35e0 msgr2=0x7fadd00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.052+0000 7fadd7d38640 1 -- 172.21.15.105:0/4201330332 shutdown_connections 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.052+0000 7fadd7d38640 1 -- 172.21.15.105:0/4201330332 wait complete. 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.053+0000 7fadd7d38640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.053+0000 7fadd7d38640 10 monclient: monmap: 2024-03-29T11:16:30.032 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 librados: starting msgr at 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 librados: starting objecter 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 Processor -- start 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 -- start start 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 librados: setting wanted keys 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 1 librados: calling monclient init 2024-03-29T11:16:30.033 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.054+0000 7fadd7d38640 10 monclient: init 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-b con 0x7fadd00af900 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-c con 0x7fadd00b4100 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient(hunting): picked mon.noname-a con 0x7fadd013ee90 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd0071b80 con 0x7fadd013ee90 2024-03-29T11:16:30.034 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd014e600 con 0x7fadd00af900 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00717a0 con 0x7fadd00b4100 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.055+0000 7fadd7d38640 10 monclient(hunting): authenticate will time out at 1264.331645s 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadd52d3640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fadd00b4100 0x7fadd01db460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:53042/0 (socket says 172.21.15.105:53042) 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadd52d3640 1 -- 172.21.15.105:0/1235129783 learned_addr learned my addr 172.21.15.105:0/1235129783 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 client.?.objecter ms_handle_connect 0x7fadd00b4100 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492343632 0 0) 0x7fadd00717a0 con 0x7fadd00b4100 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:16:30.035 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.056+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb4004eb0 con 0x7fadd00b4100 2024-03-29T11:16:30.036 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.059+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311390938 0 0) 0x7fadb4004eb0 con 0x7fadd00b4100 2024-03-29T11:16:30.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.059+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadd00717a0 con 0x7fadd00b4100 2024-03-29T11:16:30.037 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.059+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fadc8001cb0 con 0x7fadd00b4100 2024-03-29T11:16:30.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.040 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 10 monclient(hunting): dump: 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 47812612 0 0) 0x7fadd00717a0 con 0x7fadd00b4100 2024-03-29T11:16:30.041 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 >> v1:172.21.15.110:6789/0 conn(0x7fadd00af900 legacy=0x7fadd0071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.060+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 >> v1:172.21.15.105:6789/0 conn(0x7fadd013ee90 legacy=0x7fadd01dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 1 monclient: found mon.c 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd01421d0 con 0x7fadd00b4100 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 10 monclient: _renew_subs 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd0142480 con 0x7fadd00b4100 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:30.042 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 10 monclient: _renew_subs 2024-03-29T11:16:30.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd01e7120 con 0x7fadd00b4100 2024-03-29T11:16:30.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadd7d38640 1 librados: init done 2024-03-29T11:16:30.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 client.4586.objecter ms_handle_connect 0x7fadd013ee90 2024-03-29T11:16:30.043 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.061+0000 7fadc67fc640 10 client.4586.objecter resend_mon_ops 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadc8001e10 con 0x7fadd00b4100 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fadc8004fb0 con 0x7fadd00b4100 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc67fc640 10 monclient: dump: 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:30.045 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadc77fe640 4 set_mon_vals no callback set 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.063+0000 7fadd7d38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fadc801af80 con 0x7fadd00b4100 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fadc8084e90 con 0x7fadd00b4100 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 10 client.4586.objecter ms_dispatch 0x7fadd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:30.046 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.064+0000 7fadc67fc640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:16:30.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.065+0000 7fadd7d38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.065+0000 7fadd7d38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.065+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb8005080 con 0x7fadd00b4100 2024-03-29T11:16:30.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.065+0000 7fadd5ad4640 10 monclient: get_auth_request con 0x7fadb406dc90 auth_method 0 2024-03-29T11:16:30.049 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.065+0000 7fadc67fc640 10 client.4586.objecter ms_handle_connect 0x7fadb406dc90 2024-03-29T11:16:30.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.073+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fadc80508e0 con 0x7fadd00b4100 2024-03-29T11:16:30.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.073+0000 7fadc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.051 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.073+0000 7fadc67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.405+0000 7fadd7d38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.405+0000 7fadd7d38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.405+0000 7fadd7d38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.405+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadb8005330 con 0x7fadd00b4100 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.406+0000 7fadc67fc640 1 -- 172.21.15.105:0/1235129783 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fadc8052810 con 0x7fadd00b4100 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.406+0000 7fadc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:30.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.406+0000 7fadc67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:30.384 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:30.384 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:30.387 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.410+0000 7fadd7d38640 10 librados: watch_flush enter 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 10 librados: watch_flush exit 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 >> v1:172.21.15.110:6816/1087970004 conn(0x7fadb406dc90 legacy=0x7fadb4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 10 monclient: shutdown 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:30.388 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 >> v1:172.21.15.105:6790/0 conn(0x7fadd00b4100 legacy=0x7fadd01db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 shutdown_connections 2024-03-29T11:16:30.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.411+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 >> 172.21.15.105:0/1235129783 conn(0x7fadd00b35e0 msgr2=0x7fadd00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:30.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.412+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 shutdown_connections 2024-03-29T11:16:30.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.412+0000 7fadd7d38640 1 -- 172.21.15.105:0/1235129783 wait complete. 2024-03-29T11:16:30.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:30.412+0000 7fadd7d38640 1 librados: shutdown 2024-03-29T11:16:30.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:37.212 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.387+0000 7f187f474640 10 monclient: get_monmap_and_config 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.387+0000 7f187f474640 10 monclient: build_initial_monmap 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.387+0000 7f187f474640 10 monclient: monmap: 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:37.388725+0000 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:37.388725+0000 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.365 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:37.366 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:37.366 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:37.366 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.388+0000 7f187f474640 1 Processor -- start 2024-03-29T11:16:37.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.388+0000 7f187f474640 1 -- start start 2024-03-29T11:16:37.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.388+0000 7f187f474640 10 monclient: init 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 10 monclient(hunting): picked mon.noname-a con 0x7f18780b4920 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 10 monclient(hunting): picked mon.noname-c con 0x7f18780b4200 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 10 monclient(hunting): picked mon.noname-b con 0x7f187813f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.389+0000 7f187f474640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780b0930 con 0x7f18780b4920 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.390+0000 7f187f474640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780af460 con 0x7f18780b4200 2024-03-29T11:16:37.371 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.390+0000 7f187f474640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780aeda0 con 0x7f187813f9a0 2024-03-29T11:16:37.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.390+0000 7f187f474640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:37.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.390+0000 7f187f474640 10 monclient(hunting): authenticate will time out at 1271.666003s 2024-03-29T11:16:37.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.393+0000 7f187dc71640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f18780b4200 0x7f18780b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:38977/0 (socket says 172.21.15.105:38977) 2024-03-29T11:16:37.372 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.393+0000 7f187dc71640 1 -- 172.21.15.105:0/524793161 learned_addr learned my addr 172.21.15.105:0/524793161 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449137171 0 0) 0x7f18780aeda0 con 0x7f187813f9a0 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1864003520 con 0x7f187813f9a0 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424624304 0 0) 0x7f18780af460 con 0x7f18780b4200 2024-03-29T11:16:37.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:16:37.376 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.396+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18640048a0 con 0x7f18780b4200 2024-03-29T11:16:37.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371604186 0 0) 0x7f1864003520 con 0x7f187813f9a0 2024-03-29T11:16:37.378 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 >> v1:172.21.15.110:6789/0 conn(0x7f18780b4200 legacy=0x7f18780b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 >> v1:172.21.15.105:6789/0 conn(0x7f18780b4920 legacy=0x7f187813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: _finish_hunting 0 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 monclient: found mon.noname-b 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780af980 con 0x7f187813f9a0 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: _finish_auth 0 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: _check_auth_tickets 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f18700019f0 con 0x7f187813f9a0 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: dump: 2024-03-29T11:16:37.379 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187f474640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187f474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1870001b50 con 0x7f187813f9a0 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187f474640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:37.380 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187f474640 4 set_mon_vals no callback set 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 1 -- 172.21.15.105:0/524793161 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1870001f60 con 0x7f187813f9a0 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.397+0000 7f187d470640 10 monclient: dump: 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.381 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.398+0000 7f187f474640 10 monclient: shutdown 2024-03-29T11:16:37.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.398+0000 7f187f474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:37.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.398+0000 7f187f474640 1 -- 172.21.15.105:0/524793161 >> v1:172.21.15.105:6790/0 conn(0x7f187813f9a0 legacy=0x7f187814d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.401+0000 7f187f474640 1 -- 172.21.15.105:0/524793161 shutdown_connections 2024-03-29T11:16:37.382 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.401+0000 7f187f474640 1 -- 172.21.15.105:0/524793161 >> 172.21.15.105:0/524793161 conn(0x7f187806d230 msgr2=0x7f1878066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.402+0000 7f187f474640 1 -- 172.21.15.105:0/524793161 shutdown_connections 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.403+0000 7f187f474640 1 -- 172.21.15.105:0/524793161 wait complete. 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.403+0000 7f187f474640 10 monclient: build_initial_monmap 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.403+0000 7f187f474640 10 monclient: monmap: 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:37.383 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 librados: starting msgr at 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 librados: starting objecter 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 Processor -- start 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 -- start start 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 librados: setting wanted keys 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 1 librados: calling monclient init 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.404+0000 7f187f474640 10 monclient: init 2024-03-29T11:16:37.384 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient(hunting): picked mon.noname-a con 0x7f18780b4200 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient(hunting): picked mon.noname-b con 0x7f18780b4920 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient(hunting): picked mon.noname-c con 0x7f187813f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18781e4c20 con 0x7f18780b4200 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780ab630 con 0x7f18780b4920 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18781d8240 con 0x7f187813f9a0 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:37.385 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.405+0000 7f187f474640 10 monclient(hunting): authenticate will time out at 1271.681267s 2024-03-29T11:16:37.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.408+0000 7f187ec73640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f187813f9a0 0x7f18781d7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:57659/0 (socket says 172.21.15.105:57659) 2024-03-29T11:16:37.386 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.408+0000 7f187ec73640 1 -- 172.21.15.105:0/1557201242 learned_addr learned my addr 172.21.15.105:0/1557201242 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 client.?.objecter ms_handle_connect 0x7f187813f9a0 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453386443 0 0) 0x7f18781d8240 con 0x7f187813f9a0 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c004eb0 con 0x7f187813f9a0 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 client.?.objecter ms_handle_connect 0x7f18780b4200 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:37.389 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554996341 0 0) 0x7f18781e4c20 con 0x7f18780b4200 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.409+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18781d8240 con 0x7f18780b4200 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698576603 0 0) 0x7f18781d8240 con 0x7f18780b4200 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18781e4c20 con 0x7f18780b4200 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1874001510 con 0x7f18780b4200 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 10 monclient(hunting): dump: 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.390 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.410+0000 7f186effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393777501 0 0) 0x7f185c004eb0 con 0x7f187813f9a0 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18781d8240 con 0x7f187813f9a0 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1870001f60 con 0x7f187813f9a0 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 10 monclient(hunting): dump: 2024-03-29T11:16:37.391 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 440112826 0 0) 0x7f18781e4c20 con 0x7f18780b4200 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 >> v1:172.21.15.105:6790/0 conn(0x7f187813f9a0 legacy=0x7f18781d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.392 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.411+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 >> v1:172.21.15.110:6789/0 conn(0x7f18780b4920 legacy=0x7f18781d7420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:37.393 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:37.393 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 1 monclient: found mon.a 2024-03-29T11:16:37.393 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:37.393 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:37.393 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780ab8a0 con 0x7f18780b4200 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f186effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 10 monclient: _renew_subs 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18781e3d10 con 0x7f18780b4200 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 10 monclient: _renew_subs 2024-03-29T11:16:37.394 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18780a9900 con 0x7f18780b4200 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.412+0000 7f187f474640 1 librados: init done 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.414+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1874001a20 con 0x7f18780b4200 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.414+0000 7f186effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.414+0000 7f187cc6f640 4 set_mon_vals no callback set 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.414+0000 7f187f474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1874001d70 con 0x7f18780b4200 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f186effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f186effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f186effd640 10 monclient: dump: 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:37.395 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f1874019780 con 0x7f18780b4200 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.415+0000 7f187dc71640 10 monclient: get_auth_request con 0x7f185c06df50 auth_method 0 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f1874083810 con 0x7f18780b4200 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 10 client.4515.objecter ms_dispatch 0x7f1878000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:37.396 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f187f474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f187f474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860005080 con 0x7f18780b4200 2024-03-29T11:16:37.397 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.416+0000 7f186effd640 10 client.4515.objecter ms_handle_connect 0x7f185c06df50 2024-03-29T11:16:37.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.428+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f187404f170 con 0x7f18780b4200 2024-03-29T11:16:37.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.428+0000 7f186effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:37.406 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.428+0000 7f186effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:37.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.761+0000 7f187f474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.761+0000 7f187f474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.761+0000 7f187f474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.761+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1860005330 con 0x7f18780b4200 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.762+0000 7f186effd640 1 -- 172.21.15.105:0/1557201242 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f18740510a0 con 0x7f18780b4200 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.762+0000 7f186effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.762+0000 7f186effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:37.740 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:37.740 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:37.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.765+0000 7f187f474640 10 librados: watch_flush enter 2024-03-29T11:16:37.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.765+0000 7f187f474640 10 librados: watch_flush exit 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.765+0000 7f187f474640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.765+0000 7f187f474640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 >> v1:172.21.15.110:6816/1087970004 conn(0x7f185c06df50 legacy=0x7f185c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 10 monclient: shutdown 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 >> v1:172.21.15.105:6789/0 conn(0x7f18780b4200 legacy=0x7f18781e4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 shutdown_connections 2024-03-29T11:16:37.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.766+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 >> 172.21.15.105:0/1557201242 conn(0x7f187806d230 msgr2=0x7f1878069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:37.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.769+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 shutdown_connections 2024-03-29T11:16:37.747 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.769+0000 7f187f474640 1 -- 172.21.15.105:0/1557201242 wait complete. 2024-03-29T11:16:37.747 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:37.769+0000 7f187f474640 1 librados: shutdown 2024-03-29T11:16:37.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:44.569 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:44.714 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.737+0000 7fde65704640 10 monclient: get_monmap_and_config 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.737+0000 7fde65704640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.737+0000 7fde65704640 10 monclient: monmap: 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:44.738430+0000 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:44.738430+0000 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:44.715 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.738+0000 7fde65704640 1 Processor -- start 2024-03-29T11:16:44.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 1 -- start start 2024-03-29T11:16:44.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient: init 2024-03-29T11:16:44.716 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient(hunting): picked mon.noname-b con 0x7fde6006d9d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient(hunting): picked mon.noname-a con 0x7fde600b0860 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient(hunting): picked mon.noname-c con 0x7fde600af8e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600aed10 con 0x7fde600b0860 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600afd50 con 0x7fde600af8e0 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600ab1b0 con 0x7fde6006d9d0 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.717 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.739+0000 7fde65704640 10 monclient(hunting): authenticate will time out at 1279.015842s 2024-03-29T11:16:44.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5e7fc640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fde600b0860 0x7fde60146a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:55827/0 (socket says 172.21.15.105:55827) 2024-03-29T11:16:44.723 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5e7fc640 1 -- 172.21.15.105:0/1068984740 learned_addr learned my addr 172.21.15.105:0/1068984740 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529230564 0 0) 0x7fde600afd50 con 0x7fde600af8e0 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde48003520 con 0x7fde600af8e0 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321639696 0 0) 0x7fde600aed10 con 0x7fde600b0860 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.725 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.746+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde600afd50 con 0x7fde600b0860 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782835699 0 0) 0x7fde600afd50 con 0x7fde600b0860 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 >> v1:172.21.15.105:6790/0 conn(0x7fde6006d9d0 legacy=0x7fde600af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 >> v1:172.21.15.110:6789/0 conn(0x7fde600af8e0 legacy=0x7fde6014a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 monclient: found mon.noname-a 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.726 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde600ab490 con 0x7fde600b0860 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fde54001670 con 0x7fde600b0860 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: dump: 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.728 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde65704640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde65704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde540017d0 con 0x7fde600b0860 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 1 -- 172.21.15.105:0/1068984740 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fde54001ba0 con 0x7fde600b0860 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:44.729 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde5dffb640 10 monclient: dump: 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.730 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde65704640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.747+0000 7fde65704640 4 set_mon_vals no callback set 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.748+0000 7fde65704640 10 monclient: shutdown 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.749+0000 7fde65704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.749+0000 7fde65704640 1 -- 172.21.15.105:0/1068984740 >> v1:172.21.15.105:6789/0 conn(0x7fde600b0860 legacy=0x7fde60146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.749+0000 7fde65704640 1 -- 172.21.15.105:0/1068984740 shutdown_connections 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.749+0000 7fde65704640 1 -- 172.21.15.105:0/1068984740 >> 172.21.15.105:0/1068984740 conn(0x7fde6006b660 msgr2=0x7fde6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:44.731 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.752+0000 7fde65704640 1 -- 172.21.15.105:0/1068984740 shutdown_connections 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.753+0000 7fde65704640 1 -- 172.21.15.105:0/1068984740 wait complete. 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.754+0000 7fde65704640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.754+0000 7fde65704640 10 monclient: monmap: 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:44.733 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.754+0000 7fde65704640 1 librados: starting msgr at 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.754+0000 7fde65704640 1 librados: starting objecter 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 Processor -- start 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 -- start start 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 librados: setting wanted keys 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 librados: calling monclient init 2024-03-29T11:16:44.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient: init 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient(hunting): picked mon.noname-a con 0x7fde6006d9d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient(hunting): picked mon.noname-b con 0x7fde600af8e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient(hunting): picked mon.noname-c con 0x7fde600b0860 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6006dee0 con 0x7fde6006d9d0 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde60137630 con 0x7fde600af8e0 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6006f9c0 con 0x7fde600b0860 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.755+0000 7fde65704640 10 monclient(hunting): authenticate will time out at 1279.031960s 2024-03-29T11:16:44.737 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.760+0000 7fde5effd640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fde6006d9d0 0x7fde6006e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:56055/0 (socket says 172.21.15.105:56055) 2024-03-29T11:16:44.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.760+0000 7fde5effd640 1 -- 172.21.15.105:0/2814920100 learned_addr learned my addr 172.21.15.105:0/2814920100 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:44.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 client.?.objecter ms_handle_connect 0x7fde600b0860 2024-03-29T11:16:44.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188116004 0 0) 0x7fde6006f9c0 con 0x7fde600b0860 2024-03-29T11:16:44.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:16:44.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.761+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c004eb0 con 0x7fde600b0860 2024-03-29T11:16:44.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.763+0000 7fde477fe640 10 client.?.objecter ms_handle_connect 0x7fde6006d9d0 2024-03-29T11:16:44.740 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.763+0000 7fde477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.763+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707883555 0 0) 0x7fde4c004eb0 con 0x7fde600b0860 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.763+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde6006f9c0 con 0x7fde600b0860 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fde58001df0 con 0x7fde600b0860 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): dump: 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.741 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279459656 0 0) 0x7fde6006dee0 con 0x7fde6006d9d0 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:16:44.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c004eb0 con 0x7fde6006d9d0 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3637170119 0 0) 0x7fde4c004eb0 con 0x7fde6006d9d0 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde6006dee0 con 0x7fde6006d9d0 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fde500018f0 con 0x7fde6006d9d0 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient(hunting): dump: 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.744 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2460441278 0 0) 0x7fde6006f9c0 con 0x7fde600b0860 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 >> v1:172.21.15.110:6789/0 conn(0x7fde600af8e0 legacy=0x7fde6006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 >> v1:172.21.15.105:6789/0 conn(0x7fde6006d9d0 legacy=0x7fde6006e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 monclient: found mon.c 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6006fca0 con 0x7fde600b0860 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.764+0000 7fde477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde58001f50 con 0x7fde600b0860 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fde58003dc0 con 0x7fde600b0860 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde477fe640 10 monclient: dump: 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:44.746 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:44.747 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:44.747 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:44.747 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:44.747 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde5d7fa640 4 set_mon_vals no callback set 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 10 monclient: _renew_subs 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6006ff80 con 0x7fde600b0860 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 10 monclient: _renew_subs 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6008bc40 con 0x7fde600b0860 2024-03-29T11:16:44.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.765+0000 7fde65704640 1 librados: init done 2024-03-29T11:16:44.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.767+0000 7fde65704640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.767+0000 7fde65704640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.767+0000 7fde65704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:44.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.767+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60075f10 con 0x7fde600b0860 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.774+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fde58019d90 con 0x7fde600b0860 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fde58083c00 con 0x7fde600b0860 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde477fe640 10 client.4595.objecter ms_dispatch 0x7fde60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde5effd640 10 monclient: get_auth_request con 0x7fde4c06dd00 auth_method 0 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde477fe640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde477fe640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.775+0000 7fde477fe640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:16:44.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.776+0000 7fde477fe640 10 client.4595.objecter ms_handle_connect 0x7fde4c06dd00 2024-03-29T11:16:44.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.779+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde5804f4e0 con 0x7fde600b0860 2024-03-29T11:16:44.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.779+0000 7fde477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:44.779+0000 7fde477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:45.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde65704640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde65704640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde65704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde60072330 con 0x7fde600b0860 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde477fe640 1 -- 172.21.15.105:0/2814920100 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fde58051410 con 0x7fde600b0860 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.105+0000 7fde477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:45.083 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:45.084 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:45.086 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 10 librados: watch_flush enter 2024-03-29T11:16:45.086 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 10 librados: watch_flush exit 2024-03-29T11:16:45.086 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:16:45.086 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:16:45.086 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 >> v1:172.21.15.110:6816/1087970004 conn(0x7fde4c06dd00 legacy=0x7fde4c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 10 monclient: shutdown 2024-03-29T11:16:45.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:45.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.109+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 >> v1:172.21.15.105:6790/0 conn(0x7fde600b0860 legacy=0x7fde60147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.110+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 shutdown_connections 2024-03-29T11:16:45.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.110+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 >> 172.21.15.105:0/2814920100 conn(0x7fde6006b660 msgr2=0x7fde6006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:45.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.114+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 shutdown_connections 2024-03-29T11:16:45.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.114+0000 7fde65704640 1 -- 172.21.15.105:0/2814920100 wait complete. 2024-03-29T11:16:45.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:45.114+0000 7fde65704640 1 librados: shutdown 2024-03-29T11:16:45.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:51.917 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.091+0000 7f383cd8b640 10 monclient: get_monmap_and_config 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.091+0000 7f383cd8b640 10 monclient: build_initial_monmap 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.091+0000 7f383cd8b640 10 monclient: monmap: 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:52.092755+0000 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:52.092755+0000 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.069 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:52.070 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:52.070 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:52.070 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.072 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.094+0000 7f383cd8b640 1 Processor -- start 2024-03-29T11:16:52.072 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.094+0000 7f383cd8b640 1 -- start start 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.094+0000 7f383cd8b640 10 monclient: init 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-a con 0x7f383002b4c0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-b con 0x7f383000b8a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:52.073 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-c con 0x7f383002a5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:52.074 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383002c5c0 con 0x7f383002b4c0 2024-03-29T11:16:52.074 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300c11a0 con 0x7f383002a5a0 2024-03-29T11:16:52.074 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383002c830 con 0x7f383000b8a0 2024-03-29T11:16:52.074 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:52.074 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.095+0000 7f383cd8b640 10 monclient(hunting): authenticate will time out at 1286.371964s 2024-03-29T11:16:52.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.100+0000 7f38377fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f383002b4c0 0x7f3830006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46067/0 (socket says 172.21.15.105:46067) 2024-03-29T11:16:52.077 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.100+0000 7f38377fe640 1 -- 172.21.15.105:0/4032226646 learned_addr learned my addr 172.21.15.105:0/4032226646 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:52.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897697899 0 0) 0x7f383002c5c0 con 0x7f383002b4c0 2024-03-29T11:16:52.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.078 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.101+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003520 con 0x7f383002b4c0 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.102+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353204003 0 0) 0x7f3828003520 con 0x7f383002b4c0 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.102+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 >> v1:172.21.15.105:6790/0 conn(0x7f383000b8a0 legacy=0x7f38300c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:52.079 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.102+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 >> v1:172.21.15.110:6789/0 conn(0x7f383002a5a0 legacy=0x7f38300c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 1 monclient: found mon.noname-a 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383002afa0 con 0x7f383002b4c0 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:52.080 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f382c001690 con 0x7f383002b4c0 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f38367fc640 10 monclient: dump: 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:52.081 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f383cd8b640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.103+0000 7f383cd8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f382c0017f0 con 0x7f383002b4c0 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 1 -- 172.21.15.105:0/4032226646 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f382c001c40 con 0x7f383002b4c0 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:52.082 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f38367fc640 10 monclient: dump: 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f383cd8b640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.104+0000 7f383cd8b640 4 set_mon_vals no callback set 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.105+0000 7f383cd8b640 10 monclient: shutdown 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.105+0000 7f383cd8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:52.083 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.105+0000 7f383cd8b640 1 -- 172.21.15.105:0/4032226646 >> v1:172.21.15.105:6789/0 conn(0x7f383002b4c0 legacy=0x7f3830006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.085 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.108+0000 7f383cd8b640 1 -- 172.21.15.105:0/4032226646 shutdown_connections 2024-03-29T11:16:52.085 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.108+0000 7f383cd8b640 1 -- 172.21.15.105:0/4032226646 >> 172.21.15.105:0/4032226646 conn(0x7f38300058f0 msgr2=0x7f3830005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:52.087 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.109+0000 7f383cd8b640 1 -- 172.21.15.105:0/4032226646 shutdown_connections 2024-03-29T11:16:52.089 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.112+0000 7f383cd8b640 1 -- 172.21.15.105:0/4032226646 wait complete. 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.113+0000 7f383cd8b640 10 monclient: build_initial_monmap 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.113+0000 7f383cd8b640 10 monclient: monmap: 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:52.090 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.113+0000 7f383cd8b640 1 librados: starting msgr at 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.113+0000 7f383cd8b640 1 librados: starting objecter 2024-03-29T11:16:52.091 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.113+0000 7f383cd8b640 1 Processor -- start 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.114+0000 7f383cd8b640 1 -- start start 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.114+0000 7f383cd8b640 1 librados: setting wanted keys 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.114+0000 7f383cd8b640 1 librados: calling monclient init 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.114+0000 7f383cd8b640 10 monclient: init 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:52.097 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-b con 0x7f383000b8a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-a con 0x7f383002a5a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient(hunting): picked mon.noname-c con 0x7f383002b4c0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300c2c50 con 0x7f383002a5a0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300c3cd0 con 0x7f383000b8a0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38301653d0 con 0x7f383002b4c0 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:52.098 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.115+0000 7f383cd8b640 10 monclient(hunting): authenticate will time out at 1286.391252s 2024-03-29T11:16:52.099 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.121+0000 7f38377fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f383000b8a0 0x7f38300c35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:52601/0 (socket says 172.21.15.105:52601) 2024-03-29T11:16:52.099 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.121+0000 7f38377fe640 1 -- 172.21.15.105:0/4181074048 learned_addr learned my addr 172.21.15.105:0/4181074048 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:52.099 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.121+0000 7f3823fff640 10 client.?.objecter ms_handle_connect 0x7f383000b8a0 2024-03-29T11:16:52.099 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.121+0000 7f3823fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:52.100 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 client.?.objecter ms_handle_connect 0x7f383002a5a0 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062047954 0 0) 0x7f38300c2c50 con 0x7f383002a5a0 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.122+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381c004eb0 con 0x7f383002a5a0 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635733218 0 0) 0x7f38300c3cd0 con 0x7f383000b8a0 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:16:52.101 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38300c2c50 con 0x7f383000b8a0 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.123+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1266660512 0 0) 0x7f381c004eb0 con 0x7f383002a5a0 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38300c3cd0 con 0x7f383002a5a0 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f38240017d0 con 0x7f383002a5a0 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): dump: 2024-03-29T11:16:52.103 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2152341296 0 0) 0x7f38300c2c50 con 0x7f383000b8a0 2024-03-29T11:16:52.104 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f381c004eb0 con 0x7f383000b8a0 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 client.?.objecter ms_handle_connect 0x7f383002b4c0 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f382c001bc0 con 0x7f383000b8a0 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.124+0000 7f3823fff640 10 monclient(hunting): dump: 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:52.105 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:52.106 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.126+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2970713376 0 0) 0x7f381c004eb0 con 0x7f383000b8a0 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 >> v1:172.21.15.105:6790/0 conn(0x7f383002b4c0 legacy=0x7f3830164cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 >> v1:172.21.15.105:6789/0 conn(0x7f383002a5a0 legacy=0x7f38301645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 1 monclient: found mon.b 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300c4520 con 0x7f383000b8a0 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f3823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:52.107 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 10 monclient: _renew_subs 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38300c2ec0 con 0x7f383000b8a0 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 10 monclient: _renew_subs 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38300c6170 con 0x7f383000b8a0 2024-03-29T11:16:52.108 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.127+0000 7f383cd8b640 1 librados: init done 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.128+0000 7f3821ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.128+0000 7f3821ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.128+0000 7f3821ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.128+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830004cd0 con 0x7f383000b8a0 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f382c004bf0 con 0x7f383000b8a0 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:52.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f382c001d80 con 0x7f383000b8a0 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3823fff640 10 monclient: dump: 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:52.110 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:52.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.131+0000 7f3835ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f382c01ad70 con 0x7f383000b8a0 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f382c084ba0 con 0x7f383000b8a0 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 10 client.4513.objecter ms_dispatch 0x7f3838000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:52.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.134+0000 7f3823fff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:16:52.113 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.136+0000 7f3836ffd640 10 monclient: get_auth_request con 0x7f381c06dff0 auth_method 0 2024-03-29T11:16:52.114 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.137+0000 7f3823fff640 10 client.4513.objecter ms_handle_connect 0x7f381c06dff0 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.138+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f382c050480 con 0x7f383000b8a0 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.138+0000 7f3823fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:52.115 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.138+0000 7f3823fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.491+0000 7f3821ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.491+0000 7f3821ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.491+0000 7f3821ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.491+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38300c6410 con 0x7f383000b8a0 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.492+0000 7f3823fff640 1 -- 172.21.15.105:0/4181074048 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f382c0523b0 con 0x7f383000b8a0 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.492+0000 7f3823fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:52.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.492+0000 7f3823fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:52.471 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:52.471 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:52.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 10 librados: watch_flush enter 2024-03-29T11:16:52.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 10 librados: watch_flush exit 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 >> v1:172.21.15.110:6816/1087970004 conn(0x7f381c06dff0 legacy=0x7f381c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 10 monclient: shutdown 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 >> v1:172.21.15.110:6789/0 conn(0x7f383000b8a0 legacy=0x7f38300c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 shutdown_connections 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.496+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 >> 172.21.15.105:0/4181074048 conn(0x7f38300058f0 msgr2=0x7f3830005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:52.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.497+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 shutdown_connections 2024-03-29T11:16:52.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.498+0000 7f3821ffb640 1 -- 172.21.15.105:0/4181074048 wait complete. 2024-03-29T11:16:52.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:52.498+0000 7f3821ffb640 1 librados: shutdown 2024-03-29T11:16:52.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:59.302 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:59.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.472+0000 7feb8b519640 10 monclient: get_monmap_and_config 2024-03-29T11:16:59.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.473+0000 7feb8b519640 10 monclient: build_initial_monmap 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.473+0000 7feb8b519640 10 monclient: monmap: 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:16:59.474067+0000 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:16:59.474067+0000 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:16:59.451 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.474+0000 7feb8b519640 1 Processor -- start 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.474+0000 7feb8b519640 1 -- start start 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.474+0000 7feb8b519640 10 monclient: init 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.475+0000 7feb8b519640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.475+0000 7feb8b519640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.475+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c038c60 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:59.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.475+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c0324b0 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:59.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c0d11e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:59.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0caf80 con 0x7feb7c038c60 2024-03-29T11:16:59.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c039850 con 0x7feb7c0324b0 2024-03-29T11:16:59.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0c78d0 con 0x7feb7c0d11e0 2024-03-29T11:16:59.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:59.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8b519640 10 monclient(hunting): authenticate will time out at 1293.754512s 2024-03-29T11:16:59.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8a517640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7feb7c038c60 0x7feb7c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:53674/0 (socket says 172.21.15.105:53674) 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.478+0000 7feb8a517640 1 -- 172.21.15.105:0/198091967 learned_addr learned my addr 172.21.15.105:0/198091967 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705184492 0 0) 0x7feb7c0caf80 con 0x7feb7c038c60 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003520 con 0x7feb7c038c60 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56707504 0 0) 0x7feb7c039850 con 0x7feb7c0324b0 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c0caf80 con 0x7feb7c0324b0 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 551172085 0 0) 0x7feb70003520 con 0x7feb7c038c60 2024-03-29T11:16:59.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.479+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 >> v1:172.21.15.105:6790/0 conn(0x7feb7c0d11e0 legacy=0x7feb7c0d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:59.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 >> v1:172.21.15.110:6789/0 conn(0x7feb7c0324b0 legacy=0x7feb7c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: _finish_hunting 0 2024-03-29T11:16:59.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 1 monclient: found mon.noname-a 2024-03-29T11:16:59.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:59.461 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0c7a60 con 0x7feb7c038c60 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: _finish_auth 0 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: _check_auth_tickets 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7feb74001690 con 0x7feb7c038c60 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.481+0000 7feb89515640 10 monclient: dump: 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.462 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb8b519640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb8b519640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb740017f0 con 0x7feb7c038c60 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 1 -- 172.21.15.105:0/198091967 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7feb74001c40 con 0x7feb7c038c60 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb89515640 10 monclient: dump: 2024-03-29T11:16:59.463 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb8b519640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.482+0000 7feb8b519640 4 set_mon_vals no callback set 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 10 monclient: shutdown 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:59.464 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 1 -- 172.21.15.105:0/198091967 >> v1:172.21.15.105:6789/0 conn(0x7feb7c038c60 legacy=0x7feb7c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 1 -- 172.21.15.105:0/198091967 shutdown_connections 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 1 -- 172.21.15.105:0/198091967 >> 172.21.15.105:0/198091967 conn(0x7feb7c005c20 msgr2=0x7feb7c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.484+0000 7feb8b519640 1 -- 172.21.15.105:0/198091967 shutdown_connections 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.485+0000 7feb8b519640 1 -- 172.21.15.105:0/198091967 wait complete. 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.486+0000 7feb8b519640 10 monclient: build_initial_monmap 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.486+0000 7feb8b519640 10 monclient: monmap: 2024-03-29T11:16:59.466 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.486+0000 7feb8b519640 1 librados: starting msgr at 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.486+0000 7feb8b519640 1 librados: starting objecter 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.486+0000 7feb8b519640 1 Processor -- start 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.488+0000 7feb8b519640 1 -- start start 2024-03-29T11:16:59.467 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.488+0000 7feb8b519640 1 librados: setting wanted keys 2024-03-29T11:16:59.468 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.488+0000 7feb8b519640 1 librados: calling monclient init 2024-03-29T11:16:59.468 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.488+0000 7feb8b519640 10 monclient: init 2024-03-29T11:16:59.468 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c0324b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c038c60 addr v1:172.21.15.110:6789/0 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c0d11e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c034a00 con 0x7feb7c0324b0 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c15c990 con 0x7feb7c038c60 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c15c620 con 0x7feb7c0d11e0 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.489+0000 7feb8b519640 10 monclient(hunting): authenticate will time out at 1293.765845s 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.490+0000 7feb8a517640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7feb7c0324b0 0x7feb7c0c3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:54954/0 (socket says 172.21.15.105:54954) 2024-03-29T11:16:59.469 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.490+0000 7feb8a517640 1 -- 172.21.15.105:0/3803743897 learned_addr learned my addr 172.21.15.105:0/3803743897 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 client.?.objecter ms_handle_connect 0x7feb7c0324b0 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676240418 0 0) 0x7feb7c034a00 con 0x7feb7c0324b0 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:16:59.470 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.492+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb7c0324b0 2024-03-29T11:16:59.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 client.?.objecter ms_handle_connect 0x7feb7c0d11e0 2024-03-29T11:16:59.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:59.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366325899 0 0) 0x7feb7c15c620 con 0x7feb7c0d11e0 2024-03-29T11:16:59.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:59.473 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c034a00 con 0x7feb7c0d11e0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1798299701 0 0) 0x7feb68004eb0 con 0x7feb7c0324b0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb7c15c620 con 0x7feb7c0324b0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7feb74001bc0 con 0x7feb7c0324b0 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): dump: 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.474 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.475 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.496+0000 7feb7affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 10 client.?.objecter ms_handle_connect 0x7feb7c038c60 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3716366426 0 0) 0x7feb7c034a00 con 0x7feb7c0d11e0 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb7c0d11e0 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423228616 0 0) 0x7feb7c15c620 con 0x7feb7c0324b0 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 >> v1:172.21.15.105:6790/0 conn(0x7feb7c0d11e0 legacy=0x7feb7c0bc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 >> v1:172.21.15.110:6789/0 conn(0x7feb7c038c60 legacy=0x7feb7c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:59.479 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 1 monclient: found mon.a 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.497+0000 7feb7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0c45f0 con 0x7feb7c0324b0 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: _finish_auth 0 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb74004bf0 con 0x7feb7c0324b0 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7feb74004f00 con 0x7feb7c0324b0 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb7affd640 10 monclient: dump: 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:16:59.480 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb88d14640 4 set_mon_vals no callback set 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 10 monclient: _renew_subs 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c0c4840 con 0x7feb7c0324b0 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 10 monclient: _renew_subs 2024-03-29T11:16:59.481 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c0bca50 con 0x7feb7c0324b0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.498+0000 7feb8b519640 1 librados: init done 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7feb7401aed0 con 0x7feb7c0324b0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb8b519640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb8b519640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb8b519640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c0c4a30 con 0x7feb7c0324b0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7feb74084cc0 con 0x7feb7c0324b0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 10 client.4533.objecter ms_dispatch 0x7feb84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:59.482 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.501+0000 7feb7affd640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.503+0000 7feb89d16640 10 monclient: get_auth_request con 0x7feb6806dbe0 auth_method 0 2024-03-29T11:16:59.483 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.504+0000 7feb7affd640 10 client.4533.objecter ms_handle_connect 0x7feb6806dbe0 2024-03-29T11:16:59.489 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.512+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb740505a0 con 0x7feb7c0324b0 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.512+0000 7feb7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:59.490 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.512+0000 7feb7affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:59.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.842+0000 7feb8b519640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.842+0000 7feb8b519640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.842+0000 7feb8b519640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:16:59.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.842+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb7c007b70 con 0x7feb7c0324b0 2024-03-29T11:16:59.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.847+0000 7feb7affd640 1 -- 172.21.15.105:0/3803743897 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7feb740524d0 con 0x7feb7c0324b0 2024-03-29T11:16:59.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.847+0000 7feb7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.847+0000 7feb7affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:59.824 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:16:59.824 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:59.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 10 librados: watch_flush enter 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 10 librados: watch_flush exit 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 >> v1:172.21.15.110:6816/1087970004 conn(0x7feb6806dbe0 legacy=0x7feb680700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 10 monclient: shutdown 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:59.828 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.850+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 >> v1:172.21.15.105:6789/0 conn(0x7feb7c0324b0 legacy=0x7feb7c0c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.851+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 shutdown_connections 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.851+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 >> 172.21.15.105:0/3803743897 conn(0x7feb7c005c20 msgr2=0x7feb7c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.851+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 shutdown_connections 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.851+0000 7feb8b519640 1 -- 172.21.15.105:0/3803743897 wait complete. 2024-03-29T11:16:59.829 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:16:59.851+0000 7feb8b519640 1 librados: shutdown 2024-03-29T11:16:59.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:06.653 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.821+0000 7fdf6b73e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.821+0000 7fdf6b73e640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.821+0000 7fdf6b73e640 10 monclient: monmap: 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:06.822781+0000 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:06.822781+0000 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.799 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.822+0000 7fdf6b73e640 1 Processor -- start 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 1 -- start start 2024-03-29T11:17:06.800 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient: init 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf641457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf6413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf640a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf640aaef0 con 0x7fdf641457f0 2024-03-29T11:17:06.801 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf64145bd0 con 0x7fdf640a9d70 2024-03-29T11:17:06.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf64145eb0 con 0x7fdf6413f650 2024-03-29T11:17:06.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.823+0000 7fdf6b73e640 10 monclient(hunting): authenticate will time out at 1301.099880s 2024-03-29T11:17:06.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.824+0000 7fdf694da640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fdf641457f0 0x7fdf6413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:42324/0 (socket says 172.21.15.105:42324) 2024-03-29T11:17:06.802 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.824+0000 7fdf694da640 1 -- 172.21.15.105:0/1615296554 learned_addr learned my addr 172.21.15.105:0/1615296554 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:06.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.824+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420314036 0 0) 0x7fdf640aaef0 con 0x7fdf641457f0 2024-03-29T11:17:06.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.824+0000 7fdf63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.824+0000 7fdf63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54003520 con 0x7fdf641457f0 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606631396 0 0) 0x7fdf64145eb0 con 0x7fdf6413f650 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf640aaef0 con 0x7fdf6413f650 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220827453 0 0) 0x7fdf64145bd0 con 0x7fdf640a9d70 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:17:06.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf64145eb0 con 0x7fdf640a9d70 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.825+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068064502 0 0) 0x7fdf54003520 con 0x7fdf641457f0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 >> v1:172.21.15.105:6790/0 conn(0x7fdf6413f650 legacy=0x7fdf640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 >> v1:172.21.15.110:6789/0 conn(0x7fdf640a9d70 legacy=0x7fdf6414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 monclient: found mon.noname-a 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf64146340 con 0x7fdf641457f0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf58001a10 con 0x7fdf641457f0 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.812 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: dump: 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf6b73e640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf6b73e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf58001b70 con 0x7fdf641457f0 2024-03-29T11:17:06.813 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf6b73e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf6b73e640 4 set_mon_vals no callback set 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 1 -- 172.21.15.105:0/1615296554 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf58001fc0 con 0x7fdf641457f0 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.826+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf63fff640 10 monclient: dump: 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.814 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 10 monclient: shutdown 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 1 -- 172.21.15.105:0/1615296554 >> v1:172.21.15.105:6789/0 conn(0x7fdf641457f0 legacy=0x7fdf6413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 1 -- 172.21.15.105:0/1615296554 shutdown_connections 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 1 -- 172.21.15.105:0/1615296554 >> 172.21.15.105:0/1615296554 conn(0x7fdf64066a00 msgr2=0x7fdf64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 1 -- 172.21.15.105:0/1615296554 shutdown_connections 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.827+0000 7fdf6b73e640 1 -- 172.21.15.105:0/1615296554 wait complete. 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.828+0000 7fdf6b73e640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.828+0000 7fdf6b73e640 10 monclient: monmap: 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.815 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 librados: starting msgr at 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 librados: starting objecter 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 Processor -- start 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 -- start start 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 librados: setting wanted keys 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 1 librados: calling monclient init 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.829+0000 7fdf6b73e640 10 monclient: init 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf640a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:06.816 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf6413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf641dfd40 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf64070e10 con 0x7fdf641dfd40 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf64071a40 con 0x7fdf640a9d70 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6406de90 con 0x7fdf6413f650 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf6b73e640 10 monclient(hunting): authenticate will time out at 1301.106143s 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf68cd9640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fdf6413f650 0x7fdf64071330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:54400/0 (socket says 172.21.15.105:54400) 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf68cd9640 1 -- 172.21.15.105:0/3847468337 learned_addr learned my addr 172.21.15.105:0/3847468337 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter ms_handle_connect 0x7fdf6413f650 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter ms_handle_connect 0x7fdf640a9d70 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter ms_handle_connect 0x7fdf641dfd40 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.830+0000 7fdf61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.817 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096720234 0 0) 0x7fdf6406de90 con 0x7fdf6413f650 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf48004eb0 con 0x7fdf6413f650 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571545258 0 0) 0x7fdf64071a40 con 0x7fdf640a9d70 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6406de90 con 0x7fdf640a9d70 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495970202 0 0) 0x7fdf64070e10 con 0x7fdf641dfd40 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.818 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf64071a40 con 0x7fdf641dfd40 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421502186 0 0) 0x7fdf48004eb0 con 0x7fdf6413f650 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf64070e10 con 0x7fdf6413f650 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf500019b0 con 0x7fdf6413f650 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.819 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.831+0000 7fdf61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847194449 0 0) 0x7fdf6406de90 con 0x7fdf640a9d70 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf48004eb0 con 0x7fdf640a9d70 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf58001e80 con 0x7fdf640a9d70 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.820 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689067952 0 0) 0x7fdf64071a40 con 0x7fdf641dfd40 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf6406de90 con 0x7fdf641dfd40 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf5c001770 con 0x7fdf641dfd40 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:06.821 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1699103374 0 0) 0x7fdf64070e10 con 0x7fdf6413f650 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 >> v1:172.21.15.110:6789/0 conn(0x7fdf640a9d70 legacy=0x7fdf640705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.822 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 >> v1:172.21.15.105:6789/0 conn(0x7fdf641dfd40 legacy=0x7fdf6413a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 monclient: found mon.c 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6406e020 con 0x7fdf6413f650 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 10 monclient: _renew_subs 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf641df300 con 0x7fdf6413f650 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 10 monclient: _renew_subs 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:06.823 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6413ae50 con 0x7fdf6413f650 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf6b73e640 1 librados: init done 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf50003610 con 0x7fdf6413f650 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fdf50003960 con 0x7fdf6413f650 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf61ffb640 10 monclient: dump: 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:06.824 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:06.825 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:06.825 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.832+0000 7fdf637fe640 4 set_mon_vals no callback set 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.834+0000 7fdf6b73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.834+0000 7fdf6b73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.834+0000 7fdf6b73e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.834+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf44005080 con 0x7fdf6413f650 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fdf50003ac0 con 0x7fdf6413f650 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fdf50084a50 con 0x7fdf6413f650 2024-03-29T11:17:06.826 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 10 client.4607.objecter ms_dispatch 0x7fdf64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf694da640 10 monclient: get_auth_request con 0x7fdf4806e080 auth_method 0 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.835+0000 7fdf61ffb640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.836+0000 7fdf61ffb640 10 client.4607.objecter ms_handle_connect 0x7fdf4806e080 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.843+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf50050330 con 0x7fdf6413f650 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.843+0000 7fdf61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.827 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:06.843+0000 7fdf61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:07.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.216+0000 7fdf6b73e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.216+0000 7fdf6b73e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.216+0000 7fdf6b73e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:07.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.216+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf44005330 con 0x7fdf6413f650 2024-03-29T11:17:07.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.216+0000 7fdf61ffb640 1 -- 172.21.15.105:0/3847468337 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fdf50052260 con 0x7fdf6413f650 2024-03-29T11:17:07.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.217+0000 7fdf61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:07.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.217+0000 7fdf61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:07.194 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:07.194 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:07.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 10 librados: watch_flush enter 2024-03-29T11:17:07.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 10 librados: watch_flush exit 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 >> v1:172.21.15.110:6816/1087970004 conn(0x7fdf4806e080 legacy=0x7fdf48070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 10 monclient: shutdown 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 >> v1:172.21.15.105:6790/0 conn(0x7fdf6413f650 legacy=0x7fdf64071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 shutdown_connections 2024-03-29T11:17:07.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.220+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 >> 172.21.15.105:0/3847468337 conn(0x7fdf64066a00 msgr2=0x7fdf640b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:07.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.221+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 shutdown_connections 2024-03-29T11:17:07.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.221+0000 7fdf6b73e640 1 -- 172.21.15.105:0/3847468337 wait complete. 2024-03-29T11:17:07.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:07.221+0000 7fdf6b73e640 1 librados: shutdown 2024-03-29T11:17:07.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:14.021 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:14.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.190+0000 7f0b2bbe6640 10 monclient: get_monmap_and_config 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.190+0000 7f0b2bbe6640 10 monclient: build_initial_monmap 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.190+0000 7f0b2bbe6640 10 monclient: monmap: 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:14.191127+0000 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:14.191127+0000 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:14.169 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.170 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.191+0000 7f0b2bbe6640 1 Processor -- start 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.191+0000 7f0b2bbe6640 1 -- start start 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.191+0000 7f0b2bbe6640 10 monclient: init 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b240b01a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b240af010 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:14.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b240ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240ae8a0 con 0x7f0b240b01a0 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240ab1d0 con 0x7f0b240af010 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240ab4b0 con 0x7f0b240ac5a0 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.192+0000 7f0b2bbe6640 10 monclient(hunting): authenticate will time out at 1308.468712s 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b2a183640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f0b240ac5a0 0x7f0b24151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:49526/0 (socket says 172.21.15.105:49526) 2024-03-29T11:17:14.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b2a183640 1 -- 172.21.15.105:0/4025380099 learned_addr learned my addr 172.21.15.105:0/4025380099 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199795450 0 0) 0x7f0b240ab4b0 con 0x7f0b240ac5a0 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 10 monclient(hunting): _negotiate 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10003520 con 0x7f0b240ac5a0 2024-03-29T11:17:14.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517377006 0 0) 0x7f0b10003520 con 0x7f0b240ac5a0 2024-03-29T11:17:14.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.193+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 >> v1:172.21.15.110:6789/0 conn(0x7f0b240af010 legacy=0x7f0b24149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:14.174 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.196+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 >> v1:172.21.15.105:6789/0 conn(0x7f0b240b01a0 legacy=0x7f0b240abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: _finish_hunting 0 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 1 monclient: found mon.noname-b 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b240ab790 con 0x7f0b240ac5a0 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: _finish_auth 0 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: _check_auth_tickets 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0b20001690 con 0x7f0b240ac5a0 2024-03-29T11:17:14.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: dump: 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b2bbe6640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:17:14.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b2bbe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b20001860 con 0x7f0b240ac5a0 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.197+0000 7f0b28980640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b28980640 1 -- 172.21.15.105:0/4025380099 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0b20001cb0 con 0x7f0b240ac5a0 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b28980640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b28980640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b28980640 10 monclient: dump: 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.178 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b2bbe6640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.198+0000 7f0b2bbe6640 4 set_mon_vals no callback set 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.199+0000 7f0b2bbe6640 10 monclient: shutdown 2024-03-29T11:17:14.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.199+0000 7f0b2bbe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.199+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/4025380099 >> v1:172.21.15.105:6790/0 conn(0x7f0b240ac5a0 legacy=0x7f0b24151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.199+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/4025380099 shutdown_connections 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.199+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/4025380099 >> 172.21.15.105:0/4025380099 conn(0x7f0b2406b660 msgr2=0x7f0b2406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.200+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/4025380099 shutdown_connections 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.200+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/4025380099 wait complete. 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.201+0000 7f0b2bbe6640 10 monclient: build_initial_monmap 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.201+0000 7f0b2bbe6640 10 monclient: monmap: 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:14.180 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 librados: starting msgr at 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 librados: starting objecter 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 Processor -- start 2024-03-29T11:17:14.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 -- start start 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 librados: setting wanted keys 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 1 librados: calling monclient init 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.202+0000 7f0b2bbe6640 10 monclient: init 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b240ac5a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b240b01a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b240715d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2413b860 con 0x7f0b240ac5a0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b241391e0 con 0x7f0b240b01a0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2413c6e0 con 0x7f0b240715d0 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:14.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.203+0000 7f0b2bbe6640 10 monclient(hunting): authenticate will time out at 1308.479196s 2024-03-29T11:17:14.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b29181640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0b240b01a0 0x7f0b241da9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:39673/0 (socket says 172.21.15.105:39673) 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b29181640 1 -- 172.21.15.105:0/3473613154 learned_addr learned my addr 172.21.15.105:0/3473613154 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531489861 0 0) 0x7f0b241391e0 con 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.206+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08004eb0 con 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662486573 0 0) 0x7f0b08004eb0 con 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b241391e0 con 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0b1c001bf0 con 0x7f0b240b01a0 2024-03-29T11:17:14.187 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:14.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1440843660 0 0) 0x7f0b241391e0 con 0x7f0b240b01a0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 >> v1:172.21.15.105:6790/0 conn(0x7f0b240715d0 legacy=0x7f0b241de100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 >> v1:172.21.15.105:6789/0 conn(0x7f0b240ac5a0 legacy=0x7f0b2413bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 monclient: found mon.b 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24139760 con 0x7f0b240b01a0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:14.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.207+0000 7f0b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 10 monclient: _renew_subs 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b241433a0 con 0x7f0b240b01a0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 10 monclient: _renew_subs 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2413a7e0 con 0x7f0b240b01a0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.208+0000 7f0b2bbe6640 1 librados: init done 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c001d50 con 0x7f0b240b01a0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f0b1c004e60 con 0x7f0b240b01a0 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:14.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1a7fc640 10 monclient: dump: 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:14.191 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:14.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b1bfff640 4 set_mon_vals no callback set 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.209+0000 7f0b0ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.210+0000 7f0b0ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.210+0000 7f0b0ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.210+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f0b1c004fc0 con 0x7f0b240b01a0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.210+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b04005080 con 0x7f0b240b01a0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b29982640 10 monclient: get_auth_request con 0x7f0b0806dc70 auth_method 0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f0b1c084e90 con 0x7f0b240b01a0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 10 client.4528.objecter ms_dispatch 0x7f0b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:17:14.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.211+0000 7f0b1a7fc640 10 client.4528.objecter ms_handle_connect 0x7f0b0806dc70 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.219+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1c050770 con 0x7f0b240b01a0 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.219+0000 7f0b1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:14.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.219+0000 7f0b1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:14.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.573+0000 7f0b0ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.573+0000 7f0b0ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.573+0000 7f0b0ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:14.551 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.573+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b04005330 con 0x7f0b240b01a0 2024-03-29T11:17:14.552 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.575+0000 7f0b1a7fc640 1 -- 172.21.15.105:0/3473613154 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f0b1c0526a0 con 0x7f0b240b01a0 2024-03-29T11:17:14.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.575+0000 7f0b1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:14.553 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.575+0000 7f0b1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:14.553 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:14.553 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.579+0000 7f0b0ffff640 10 librados: watch_flush enter 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.579+0000 7f0b0ffff640 10 librados: watch_flush exit 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.579+0000 7f0b0ffff640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.579+0000 7f0b0ffff640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 >> v1:172.21.15.110:6816/1087970004 conn(0x7f0b0806dc70 legacy=0x7f0b08070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 10 monclient: shutdown 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 >> v1:172.21.15.110:6789/0 conn(0x7f0b240b01a0 legacy=0x7f0b241da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 shutdown_connections 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 >> 172.21.15.105:0/3473613154 conn(0x7f0b2406b660 msgr2=0x7f0b240b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 shutdown_connections 2024-03-29T11:17:14.558 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 -- 172.21.15.105:0/3473613154 wait complete. 2024-03-29T11:17:14.559 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:14.580+0000 7f0b0ffff640 1 librados: shutdown 2024-03-29T11:17:14.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:21.380 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:21.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.536+0000 7fe6f58fb640 10 monclient: get_monmap_and_config 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.536+0000 7fe6f58fb640 10 monclient: build_initial_monmap 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.537+0000 7fe6f58fb640 10 monclient: monmap: 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:21.538044+0000 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:21.538044+0000 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:21.515 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.538+0000 7fe6f58fb640 1 Processor -- start 2024-03-29T11:17:21.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.538+0000 7fe6f58fb640 1 -- start start 2024-03-29T11:17:21.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.538+0000 7fe6f58fb640 10 monclient: init 2024-03-29T11:17:21.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:21.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6f01457f0 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe6f013f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6f00a9d70 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f00aaef0 con 0x7fe6f013f650 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ef7fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fe6f00a9d70 0x7fe6f014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:48026/0 (socket says 172.21.15.105:48026) 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ef7fe640 1 -- 172.21.15.105:0/88269647 learned_addr learned my addr 172.21.15.105:0/88269647 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0145bd0 con 0x7fe6f01457f0 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0145eb0 con 0x7fe6f00a9d70 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6f58fb640 10 monclient(hunting): authenticate will time out at 1315.815767s 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888190426 0 0) 0x7fe6f0145eb0 con 0x7fe6f00a9d70 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.539+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6dc003520 con 0x7fe6f00a9d70 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.540+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377700018 0 0) 0x7fe6dc003520 con 0x7fe6f00a9d70 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.540+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 >> v1:172.21.15.110:6789/0 conn(0x7fe6f01457f0 legacy=0x7fe6f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:21.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.540+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 >> v1:172.21.15.105:6789/0 conn(0x7fe6f013f650 legacy=0x7fe6f00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:21.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:21.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 1 monclient: found mon.noname-b 2024-03-29T11:17:21.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:21.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:17:21.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0146340 con 0x7fe6f00a9d70 2024-03-29T11:17:21.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:21.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:21.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe6e8001a10 con 0x7fe6f00a9d70 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: dump: 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.523 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6e8001be0 con 0x7fe6f00a9d70 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 1 -- 172.21.15.105:0/88269647 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe6e8002030 con 0x7fe6f00a9d70 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6f58fb640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6f58fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6f58fb640 4 set_mon_vals no callback set 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.542+0000 7fe6ee7fc640 10 monclient: dump: 2024-03-29T11:17:21.524 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 10 monclient: shutdown 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 >> v1:172.21.15.105:6790/0 conn(0x7fe6f00a9d70 legacy=0x7fe6f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 shutdown_connections 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 >> 172.21.15.105:0/88269647 conn(0x7fe6f0066a00 msgr2=0x7fe6f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:21.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 shutdown_connections 2024-03-29T11:17:21.526 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.543+0000 7fe6f58fb640 1 -- 172.21.15.105:0/88269647 wait complete. 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.545+0000 7fe6f58fb640 10 monclient: build_initial_monmap 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.545+0000 7fe6f58fb640 10 monclient: monmap: 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:21.528 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.545+0000 7fe6f58fb640 1 librados: starting msgr at 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.545+0000 7fe6f58fb640 1 librados: starting objecter 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.545+0000 7fe6f58fb640 1 Processor -- start 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.546+0000 7fe6f58fb640 1 -- start start 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.546+0000 7fe6f58fb640 1 librados: setting wanted keys 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.546+0000 7fe6f58fb640 1 librados: calling monclient init 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.546+0000 7fe6f58fb640 10 monclient: init 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe6f00a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6f013f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:21.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6f01457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f01d7030 con 0x7fe6f00a9d70 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f00ad350 con 0x7fe6f013f650 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0142ca0 con 0x7fe6f01457f0 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6f58fb640 10 monclient(hunting): authenticate will time out at 1315.823750s 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6e7fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe6f013f650 0x7fe6f01d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:33622/0 (socket says 172.21.15.105:33622) 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.547+0000 7fe6e7fff640 1 -- 172.21.15.105:0/255760499 learned_addr learned my addr 172.21.15.105:0/255760499 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 client.?.objecter ms_handle_connect 0x7fe6f013f650 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613381962 0 0) 0x7fe6f00ad350 con 0x7fe6f013f650 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d0004eb0 con 0x7fe6f013f650 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.548+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3729378963 0 0) 0x7fe6d0004eb0 con 0x7fe6f013f650 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6d0005300 con 0x7fe6f013f650 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe6d8004040 con 0x7fe6f013f650 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 10 monclient(hunting): dump: 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:21.531 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.549+0000 7fe6ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 client.?.objecter ms_handle_connect 0x7fe6f01457f0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576061210 0 0) 0x7fe6f0142ca0 con 0x7fe6f01457f0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d00058a0 con 0x7fe6f01457f0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 client.?.objecter ms_handle_connect 0x7fe6f00a9d70 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.550+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605668808 0 0) 0x7fe6d00058a0 con 0x7fe6f01457f0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6f0142ca0 con 0x7fe6f01457f0 2024-03-29T11:17:21.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463247994 0 0) 0x7fe6f01d7030 con 0x7fe6f00a9d70 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d00058a0 con 0x7fe6f00a9d70 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381870255 0 0) 0x7fe6d0005300 con 0x7fe6f013f650 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 >> v1:172.21.15.105:6790/0 conn(0x7fe6f01457f0 legacy=0x7fe6f0137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 >> v1:172.21.15.105:6789/0 conn(0x7fe6f00a9d70 legacy=0x7fe6f00ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 monclient: found mon.b 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f013fda0 con 0x7fe6f013f650 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:17:21.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:17:21.534 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:17:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 10 monclient: _renew_subs 2024-03-29T11:17:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:21.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f00ac0f0 con 0x7fe6f013f650 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 10 monclient: _renew_subs 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f0138450 con 0x7fe6f013f650 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6f58fb640 1 librados: init done 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6d8000ed0 con 0x7fe6f013f650 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fe6d8004040 con 0x7fe6f013f650 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.551+0000 7fe6ecff9640 10 monclient: dump: 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.536 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.552+0000 7fe6f58fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.552+0000 7fe6ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fe6d8001040 con 0x7fe6f013f650 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6f58fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6f58fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0075f10 con 0x7fe6f013f650 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fe6d80845a0 con 0x7fe6f013f650 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 10 client.4534.objecter ms_dispatch 0x7fe6f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:21.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:21.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:21.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6ecff9640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:17:21.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.553+0000 7fe6eeffd640 10 monclient: get_auth_request con 0x7fe6d006e600 auth_method 0 2024-03-29T11:17:21.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.554+0000 7fe6ecff9640 10 client.4534.objecter ms_handle_connect 0x7fe6d006e600 2024-03-29T11:17:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.561+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6d804fe80 con 0x7fe6f013f650 2024-03-29T11:17:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.561+0000 7fe6ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:21.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.561+0000 7fe6ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:21.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.877+0000 7fe6f58fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.877+0000 7fe6f58fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.877+0000 7fe6f58fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:21.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.877+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6f0138700 con 0x7fe6f013f650 2024-03-29T11:17:21.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.880+0000 7fe6ecff9640 1 -- 172.21.15.105:0/255760499 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fe6d804fe80 con 0x7fe6f013f650 2024-03-29T11:17:21.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.880+0000 7fe6ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.880+0000 7fe6ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:21.857 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:21.858 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:21.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 10 librados: watch_flush enter 2024-03-29T11:17:21.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 10 librados: watch_flush exit 2024-03-29T11:17:21.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 >> v1:172.21.15.110:6816/1087970004 conn(0x7fe6d006e600 legacy=0x7fe6d0070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 10 monclient: shutdown 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 >> v1:172.21.15.110:6789/0 conn(0x7fe6f013f650 legacy=0x7fe6f01d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 shutdown_connections 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 >> 172.21.15.105:0/255760499 conn(0x7fe6f0066a00 msgr2=0x7fe6f0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 shutdown_connections 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 -- 172.21.15.105:0/255760499 wait complete. 2024-03-29T11:17:21.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:21.884+0000 7fe6f58fb640 1 librados: shutdown 2024-03-29T11:17:21.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:28.687 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.855+0000 7f504ce8f640 10 monclient: get_monmap_and_config 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.855+0000 7f504ce8f640 10 monclient: build_initial_monmap 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.855+0000 7f504ce8f640 10 monclient: monmap: 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:28.856974+0000 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:28.856974+0000 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:28.833 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.857+0000 7f504ce8f640 1 Processor -- start 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.857+0000 7f504ce8f640 1 -- start start 2024-03-29T11:17:28.834 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.857+0000 7f504ce8f640 10 monclient: init 2024-03-29T11:17:28.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:28.835 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-c con 0x7f50480b1f60 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-a con 0x7f5048071eb0 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-b con 0x7f5048143d10 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504806ef80 con 0x7f5048071eb0 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504806f260 con 0x7f50480b1f60 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480b40c0 con 0x7f5048143d10 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:28.836 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.858+0000 7f504ce8f640 10 monclient(hunting): authenticate will time out at 1323.134670s 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f5046d9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f5048143d10 0x7f5048146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:60700/0 (socket says 172.21.15.105:60700) 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f5046d9d640 1 -- 172.21.15.105:0/2391810769 learned_addr learned my addr 172.21.15.105:0/2391810769 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889244015 0 0) 0x7f50480b40c0 con 0x7f5048143d10 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.859+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c003520 con 0x7f5048143d10 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.860+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850390357 0 0) 0x7f502c003520 con 0x7f5048143d10 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.860+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 >> v1:172.21.15.110:6789/0 conn(0x7f50480b1f60 legacy=0x7f5048143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:28.837 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.860+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 >> v1:172.21.15.105:6789/0 conn(0x7f5048071eb0 legacy=0x7f5048146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:28.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:28.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 1 monclient: found mon.noname-b 2024-03-29T11:17:28.838 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048155d70 con 0x7f5048143d10 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: _finish_auth 0 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5040001a10 con 0x7f5048143d10 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: dump: 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.839 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504ce8f640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504ce8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5040001be0 con 0x7f5048143d10 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504ce8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504ce8f640 4 set_mon_vals no callback set 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 1 -- 172.21.15.105:0/2391810769 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5040002030 con 0x7f5048143d10 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.861+0000 7f504559a640 10 monclient: dump: 2024-03-29T11:17:28.840 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:28.841 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 10 monclient: shutdown 2024-03-29T11:17:28.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:28.844 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 1 -- 172.21.15.105:0/2391810769 >> v1:172.21.15.105:6790/0 conn(0x7f5048143d10 legacy=0x7f5048146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 1 -- 172.21.15.105:0/2391810769 shutdown_connections 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 1 -- 172.21.15.105:0/2391810769 >> 172.21.15.105:0/2391810769 conn(0x7f5048066a00 msgr2=0x7f5048066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 1 -- 172.21.15.105:0/2391810769 shutdown_connections 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.862+0000 7f504ce8f640 1 -- 172.21.15.105:0/2391810769 wait complete. 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.863+0000 7f504ce8f640 10 monclient: build_initial_monmap 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.863+0000 7f504ce8f640 10 monclient: monmap: 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:28.845 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 librados: starting msgr at 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 librados: starting objecter 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 Processor -- start 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 -- start start 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 librados: setting wanted keys 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 1 librados: calling monclient init 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.864+0000 7f504ce8f640 10 monclient: init 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:28.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-c con 0x7f50480b1f60 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-b con 0x7f5048143d10 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient(hunting): picked mon.noname-a con 0x7f5048140360 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480ac130 con 0x7f5048140360 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50481407d0 con 0x7f5048143d10 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504806fa70 con 0x7f50480b1f60 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504ce8f640 10 monclient(hunting): authenticate will time out at 1323.141426s 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504659c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f50480b1f60 0x7f50480aada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:32784/0 (socket says 172.21.15.105:32784) 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.865+0000 7f504659c640 1 -- 172.21.15.105:0/3163198799 learned_addr learned my addr 172.21.15.105:0/3163198799 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 client.?.objecter ms_handle_connect 0x7f50480b1f60 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265380076 0 0) 0x7f504806fa70 con 0x7f50480b1f60 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:17:28.848 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.866+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030004eb0 con 0x7f50480b1f60 2024-03-29T11:17:28.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.870+0000 7f503effd640 10 client.?.objecter ms_handle_connect 0x7f5048143d10 2024-03-29T11:17:28.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.870+0000 7f503effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:28.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.870+0000 7f503effd640 10 client.?.objecter ms_handle_connect 0x7f5048140360 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.870+0000 7f503effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.870+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1457437302 0 0) 0x7f5030004eb0 con 0x7f50480b1f60 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f504806fa70 con 0x7f50480b1f60 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064042494 0 0) 0x7f50481407d0 con 0x7f5048143d10 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:17:28.852 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030004eb0 con 0x7f5048143d10 2024-03-29T11:17:28.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181318603 0 0) 0x7f50480ac130 con 0x7f5048140360 2024-03-29T11:17:28.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.853 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50481407d0 con 0x7f5048140360 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5038001870 con 0x7f50480b1f60 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): dump: 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:28.854 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375956163 0 0) 0x7f50481407d0 con 0x7f5048140360 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50480ac130 con 0x7f5048140360 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655201134 0 0) 0x7f504806fa70 con 0x7f50480b1f60 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.871+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 >> v1:172.21.15.110:6789/0 conn(0x7f5048143d10 legacy=0x7f504813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 >> v1:172.21.15.105:6789/0 conn(0x7f5048140360 legacy=0x7f504806ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 1 monclient: found mon.c 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480ac380 con 0x7f50480b1f60 2024-03-29T11:17:28.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 10 monclient: _finish_auth 0 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.872+0000 7f503effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 10 monclient: _renew_subs 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50480ab4b0 con 0x7f50480b1f60 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 10 monclient: _renew_subs 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50480aa270 con 0x7f50480b1f60 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f504ce8f640 1 librados: init done 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5038001dc0 con 0x7f50480b1f60 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f503effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.873+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f5038004ac0 con 0x7f50480b1f60 2024-03-29T11:17:28.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.874+0000 7f503effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.874+0000 7f503effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.874+0000 7f503effd640 10 monclient: dump: 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:28.857 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:28.858 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.875+0000 7f503ffff640 4 set_mon_vals no callback set 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.875+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f503801aa90 con 0x7f50480b1f60 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.876+0000 7f504ce8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.877+0000 7f504ce8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.877+0000 7f504ce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.877+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5024005080 con 0x7f50480b1f60 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.877+0000 7f5045d9b640 10 monclient: get_auth_request con 0x7f503006dec0 auth_method 0 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.878+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f50380849e0 con 0x7f50480b1f60 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.878+0000 7f503effd640 10 client.4622.objecter ms_dispatch 0x7f5048000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.878+0000 7f503effd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.878+0000 7f503effd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.878+0000 7f503effd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:17:28.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.880+0000 7f503effd640 10 client.4622.objecter ms_handle_connect 0x7f503006dec0 2024-03-29T11:17:28.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.885+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50380502c0 con 0x7f50480b1f60 2024-03-29T11:17:28.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.885+0000 7f503effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:28.885+0000 7f503effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:29.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.214+0000 7f504ce8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.192 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.214+0000 7f504ce8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.214+0000 7f504ce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:29.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.214+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5024005330 con 0x7f50480b1f60 2024-03-29T11:17:29.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.215+0000 7f503effd640 1 -- 172.21.15.105:0/3163198799 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f50380521f0 con 0x7f50480b1f60 2024-03-29T11:17:29.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.215+0000 7f503effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:29.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.215+0000 7f503effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:29.194 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:29.194 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:29.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 10 librados: watch_flush enter 2024-03-29T11:17:29.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 10 librados: watch_flush exit 2024-03-29T11:17:29.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:17:29.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:17:29.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 >> v1:172.21.15.110:6816/1087970004 conn(0x7f503006dec0 legacy=0x7f5030070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 10 monclient: shutdown 2024-03-29T11:17:29.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:29.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.218+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 >> v1:172.21.15.105:6790/0 conn(0x7f50480b1f60 legacy=0x7f50480aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.220+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 shutdown_connections 2024-03-29T11:17:29.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.220+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 >> 172.21.15.105:0/3163198799 conn(0x7f5048066a00 msgr2=0x7f5048066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:29.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.222+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 shutdown_connections 2024-03-29T11:17:29.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.222+0000 7f504ce8f640 1 -- 172.21.15.105:0/3163198799 wait complete. 2024-03-29T11:17:29.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:29.222+0000 7f504ce8f640 1 librados: shutdown 2024-03-29T11:17:29.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:36.023 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:36.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.190+0000 7f9635d48640 10 monclient: get_monmap_and_config 2024-03-29T11:17:36.167 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.190+0000 7f9635d48640 10 monclient: build_initial_monmap 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.190+0000 7f9635d48640 10 monclient: monmap: 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:36.191397+0000 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:36.191397+0000 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:36.168 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.194+0000 7f9635d48640 1 Processor -- start 2024-03-29T11:17:36.171 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.194+0000 7f9635d48640 1 -- start start 2024-03-29T11:17:36.172 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.194+0000 7f9635d48640 10 monclient: init 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.195+0000 7f9635d48640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.195+0000 7f9635d48640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.196+0000 7f962f7fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f96301457f0 0x7f963013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:52332/0 (socket says 172.21.15.105:52332) 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.196+0000 7f962f7fe640 1 -- 172.21.15.105:0/3601939654 learned_addr learned my addr 172.21.15.105:0/3601939654 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.196+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-b con 0x7f96301457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:36.173 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.196+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-c con 0x7f963013f650 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:36.175 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.196+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-a con 0x7f96300a9d70 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:36.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.198+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300aaef0 con 0x7f96300a9d70 2024-03-29T11:17:36.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.198+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9630145bd0 con 0x7f963013f650 2024-03-29T11:17:36.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.198+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9630145eb0 con 0x7f96301457f0 2024-03-29T11:17:36.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.198+0000 7f9635d48640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:36.176 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.198+0000 7f9635d48640 10 monclient(hunting): authenticate will time out at 1330.474850s 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741735083 0 0) 0x7f9630145eb0 con 0x7f96301457f0 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618003520 con 0x7f96301457f0 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264274997 0 0) 0x7f9618003520 con 0x7f96301457f0 2024-03-29T11:17:36.177 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.199+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 >> v1:172.21.15.110:6789/0 conn(0x7f963013f650 legacy=0x7f96300b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 >> v1:172.21.15.105:6789/0 conn(0x7f96300a9d70 legacy=0x7f963014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630146340 con 0x7f96301457f0 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9628001a10 con 0x7f96301457f0 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.179 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f962e7fc640 10 monclient: dump: 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f9635d48640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:17:36.180 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.201+0000 7f9635d48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9628001be0 con 0x7f96301457f0 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 1 -- 172.21.15.105:0/3601939654 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9628001ff0 con 0x7f96301457f0 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f962e7fc640 10 monclient: dump: 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.181 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f9635d48640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:36.182 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.203+0000 7f9635d48640 4 set_mon_vals no callback set 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.204+0000 7f9635d48640 10 monclient: shutdown 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.204+0000 7f9635d48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.204+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 >> v1:172.21.15.105:6790/0 conn(0x7f96301457f0 legacy=0x7f963013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.204+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 shutdown_connections 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.204+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 >> 172.21.15.105:0/3601939654 conn(0x7f9630066a00 msgr2=0x7f9630066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.205+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 shutdown_connections 2024-03-29T11:17:36.183 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.205+0000 7f9635d48640 1 -- 172.21.15.105:0/3601939654 wait complete. 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.206+0000 7f9635d48640 10 monclient: build_initial_monmap 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.206+0000 7f9635d48640 10 monclient: monmap: 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:36.184 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.207+0000 7f9635d48640 1 librados: starting msgr at 2024-03-29T11:17:36.185 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.207+0000 7f9635d48640 1 librados: starting objecter 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.207+0000 7f9635d48640 1 Processor -- start 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.208+0000 7f9635d48640 1 -- start start 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.208+0000 7f9635d48640 1 librados: setting wanted keys 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.208+0000 7f9635d48640 1 librados: calling monclient init 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.208+0000 7f9635d48640 10 monclient: init 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.209+0000 7f9635d48640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:36.186 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.209+0000 7f9635d48640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:36.188 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-b con 0x7f96300a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-a con 0x7f963013f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 10 monclient(hunting): picked mon.noname-c con 0x7f96301457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300adaf0 con 0x7f963013f650 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300ad710 con 0x7f96300a9d70 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300ad8a0 con 0x7f96301457f0 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:36.189 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.211+0000 7f9635d48640 10 monclient(hunting): authenticate will time out at 1330.487947s 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f962f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f96300a9d70 0x7f96301d7070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:45179/0 (socket says 172.21.15.105:45179) 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f962f7fe640 1 -- 172.21.15.105:0/4137095144 learned_addr learned my addr 172.21.15.105:0/4137095144 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f962ffff640 1 --1- 172.21.15.105:0/4137095144 >> v1:172.21.15.105:6790/0 conn(0x7f96301457f0 0x7f96300b11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:52912/0 (socket says 172.21.15.105:52912) 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 client.?.objecter ms_handle_connect 0x7f96300a9d70 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 client.?.objecter ms_handle_connect 0x7f96301457f0 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:36.190 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949222158 0 0) 0x7f96300ad710 con 0x7f96300a9d70 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.212+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f96300a9d70 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236028713 0 0) 0x7f96300ad8a0 con 0x7f96301457f0 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:17:36.191 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300ad710 con 0x7f96301457f0 2024-03-29T11:17:36.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282947983 0 0) 0x7f961c004eb0 con 0x7f96300a9d70 2024-03-29T11:17:36.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961c005490 con 0x7f96300a9d70 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f9628001fb0 con 0x7f96300a9d70 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): dump: 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:36.194 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800212132 0 0) 0x7f96300ad710 con 0x7f96301457f0 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f96301457f0 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f96240018b0 con 0x7f96301457f0 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.213+0000 7f9617fff640 10 monclient(hunting): dump: 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.195 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.214+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3560185880 0 0) 0x7f961c004eb0 con 0x7f96301457f0 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.214+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 >> v1:172.21.15.110:6789/0 conn(0x7f96300a9d70 legacy=0x7f96301d7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.214+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 >> v1:172.21.15.105:6789/0 conn(0x7f963013f650 legacy=0x7f96300b0ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 1 monclient: found mon.c 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300ae010 con 0x7f96301457f0 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:36.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 10 monclient: _renew_subs 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96301d7780 con 0x7f96301457f0 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 10 monclient: _renew_subs 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9630142250 con 0x7f96301457f0 2024-03-29T11:17:36.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.215+0000 7f9635d48640 1 librados: init done 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.217+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9624001d40 con 0x7f96301457f0 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.217+0000 7f9617fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.217+0000 7f962d7fa640 4 set_mon_vals no callback set 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9635d48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f96240018b0 con 0x7f96301457f0 2024-03-29T11:17:36.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9635d48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9635d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9610005080 con 0x7f96301457f0 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9617fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.218+0000 7f9617fff640 10 monclient: dump: 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:36.199 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.219+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f96240036b0 con 0x7f96301457f0 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f9624084650 con 0x7f96301457f0 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 10 client.4628.objecter ms_dispatch 0x7f9630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f962f7fe640 10 monclient: get_auth_request con 0x7f961c06e230 auth_method 0 2024-03-29T11:17:36.200 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.220+0000 7f9617fff640 10 client.4628.objecter ms_handle_connect 0x7f961c06e230 2024-03-29T11:17:36.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.228+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f962404ff90 con 0x7f96301457f0 2024-03-29T11:17:36.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.228+0000 7f9617fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:36.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.228+0000 7f9617fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:36.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.551+0000 7f9635d48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.551+0000 7f9635d48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.551+0000 7f9635d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:36.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.551+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9610005330 con 0x7f96301457f0 2024-03-29T11:17:36.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.554+0000 7f9617fff640 1 -- 172.21.15.105:0/4137095144 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f9624083d00 con 0x7f96301457f0 2024-03-29T11:17:36.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.555+0000 7f9617fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:36.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.555+0000 7f9617fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:36.532 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:36.532 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:36.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.558+0000 7f9635d48640 10 librados: watch_flush enter 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 10 librados: watch_flush exit 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 >> v1:172.21.15.110:6816/1087970004 conn(0x7f961c06e230 legacy=0x7f961c0706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 10 monclient: shutdown 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 >> v1:172.21.15.105:6790/0 conn(0x7f96301457f0 legacy=0x7f96300b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 shutdown_connections 2024-03-29T11:17:36.537 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 >> 172.21.15.105:0/4137095144 conn(0x7f9630066a00 msgr2=0x7f9630066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:36.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 shutdown_connections 2024-03-29T11:17:36.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.559+0000 7f9635d48640 1 -- 172.21.15.105:0/4137095144 wait complete. 2024-03-29T11:17:36.538 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:36.560+0000 7f9635d48640 1 librados: shutdown 2024-03-29T11:17:36.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:43.364 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.532+0000 7f46b12e0640 10 monclient: get_monmap_and_config 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.532+0000 7f46b12e0640 10 monclient: build_initial_monmap 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.532+0000 7f46b12e0640 10 monclient: monmap: 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:43.533652+0000 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:43.533652+0000 2024-03-29T11:17:43.511 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.533+0000 7f46b12e0640 1 Processor -- start 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.533+0000 7f46b12e0640 1 -- start start 2024-03-29T11:17:43.512 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.533+0000 7f46b12e0640 10 monclient: init 2024-03-29T11:17:43.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.534+0000 7f46b12e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:43.513 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.534+0000 7f46b12e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.534+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-b con 0x7f46ac0a9ac0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.534+0000 7f46aaffd640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f46ac0a9ac0 0x7f46ac0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:48894/0 (socket says 172.21.15.105:48894) 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.534+0000 7f46aaffd640 1 -- 172.21.15.105:0/2388325095 learned_addr learned my addr 172.21.15.105:0/2388325095 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-a con 0x7f46ac0b4100 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-c con 0x7f46ac13ee90 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0af6f0 con 0x7f46ac0b4100 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0aac80 con 0x7f46ac13ee90 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0aaf60 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:43.514 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.535+0000 7f46b12e0640 10 monclient(hunting): authenticate will time out at 1337.811785s 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247230825 0 0) 0x7f46ac0aaf60 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694003520 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290503631 0 0) 0x7f4694003520 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.515 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.537+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 >> v1:172.21.15.110:6789/0 conn(0x7f46ac13ee90 legacy=0x7f46ac149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 >> v1:172.21.15.105:6789/0 conn(0x7f46ac0b4100 legacy=0x7f46ac13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac13e970 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.518 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f46a40052f0 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: dump: 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:43.519 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46b12e0640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46b12e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a4001380 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 1 -- 172.21.15.105:0/2388325095 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f46a40017d0 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46b12e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46b12e0640 4 set_mon_vals no callback set 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.539+0000 7f46a9ffb640 10 monclient: dump: 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:43.520 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 10 monclient: shutdown 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 >> v1:172.21.15.105:6790/0 conn(0x7f46ac0a9ac0 legacy=0x7f46ac0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 shutdown_connections 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 >> 172.21.15.105:0/2388325095 conn(0x7f46ac0b35e0 msgr2=0x7f46ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:43.521 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 shutdown_connections 2024-03-29T11:17:43.522 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.540+0000 7f46b12e0640 1 -- 172.21.15.105:0/2388325095 wait complete. 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.541+0000 7f46b12e0640 10 monclient: build_initial_monmap 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.541+0000 7f46b12e0640 10 monclient: monmap: 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:43.523 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 librados: starting msgr at 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 librados: starting objecter 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 Processor -- start 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 -- start start 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 librados: setting wanted keys 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 1 librados: calling monclient init 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.542+0000 7f46b12e0640 10 monclient: init 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.543+0000 7f46b12e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.543+0000 7f46b12e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-b con 0x7f46ac0a9ac0 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-a con 0x7f46ac0b4100 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:43.524 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 10 monclient(hunting): picked mon.noname-c con 0x7f46ac13ee90 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:43.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac141cd0 con 0x7f46ac0b4100 2024-03-29T11:17:43.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac14e480 con 0x7f46ac0a9ac0 2024-03-29T11:17:43.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac140b70 con 0x7f46ac13ee90 2024-03-29T11:17:43.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:43.525 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.544+0000 7f46b12e0640 10 monclient(hunting): authenticate will time out at 1337.820822s 2024-03-29T11:17:43.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.547+0000 7f46aaffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f46ac0a9ac0 0x7f46ac13ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:36305/0 (socket says 172.21.15.105:36305) 2024-03-29T11:17:43.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.547+0000 7f46aaffd640 1 -- 172.21.15.105:0/1341578325 learned_addr learned my addr 172.21.15.105:0/1341578325 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:43.529 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 client.?.objecter ms_handle_connect 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523568255 0 0) 0x7f46ac140b70 con 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4698004eb0 con 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164160301 0 0) 0x7f4698004eb0 con 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ac140b70 con 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f46a0001750 con 0x7f46ac13ee90 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): dump: 2024-03-29T11:17:43.530 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2650855677 0 0) 0x7f46ac140b70 con 0x7f46ac13ee90 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 >> v1:172.21.15.110:6789/0 conn(0x7f46ac0a9ac0 legacy=0x7f46ac13ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.548+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 >> v1:172.21.15.105:6789/0 conn(0x7f46ac0b4100 legacy=0x7f46ac13d0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:43.531 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 1 monclient: found mon.c 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac13f670 con 0x7f46ac13ee90 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 10 monclient: _renew_subs 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac140690 con 0x7f46ac13ee90 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 10 monclient: _renew_subs 2024-03-29T11:17:43.532 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac142460 con 0x7f46ac13ee90 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.549+0000 7f46b12e0640 1 librados: init done 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.550+0000 7f46b12e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.550+0000 7f46b12e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.550+0000 7f46b12e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.533 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.550+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac075f10 con 0x7f46ac13ee90 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a0001d20 con 0x7f46ac13ee90 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f46a0001750 con 0x7f46ac13ee90 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 10 monclient: dump: 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.535 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f46a0019560 con 0x7f46ac13ee90 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46a97fa640 4 set_mon_vals no callback set 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f46a00835e0 con 0x7f46ac13ee90 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 10 client.4634.objecter ms_dispatch 0x7f46ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46937fe640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.554+0000 7f46aaffd640 10 monclient: get_auth_request con 0x7f4698072360 auth_method 0 2024-03-29T11:17:43.536 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.555+0000 7f46937fe640 10 client.4634.objecter ms_handle_connect 0x7f4698072360 2024-03-29T11:17:43.539 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.562+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46a004ed70 con 0x7f46ac13ee90 2024-03-29T11:17:43.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.562+0000 7f46937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.540 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.562+0000 7f46937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:43.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.883+0000 7f46b12e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.883+0000 7f46b12e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.883+0000 7f46b12e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:43.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.883+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46ac13f800 con 0x7f46ac13ee90 2024-03-29T11:17:43.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.884+0000 7f46937fe640 1 -- 172.21.15.105:0/1341578325 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f46a004ed70 con 0x7f46ac13ee90 2024-03-29T11:17:43.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.884+0000 7f46937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.884+0000 7f46937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:43.863 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:43.863 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:43.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 10 librados: watch_flush enter 2024-03-29T11:17:43.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 10 librados: watch_flush exit 2024-03-29T11:17:43.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:17:43.865 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 >> v1:172.21.15.110:6816/1087970004 conn(0x7f4698072360 legacy=0x7f4698074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 10 monclient: shutdown 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 >> v1:172.21.15.105:6790/0 conn(0x7f46ac13ee90 legacy=0x7f46ac1e02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 shutdown_connections 2024-03-29T11:17:43.866 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.888+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 >> 172.21.15.105:0/1341578325 conn(0x7f46ac0b35e0 msgr2=0x7f46ac0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:43.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.889+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 shutdown_connections 2024-03-29T11:17:43.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.889+0000 7f46b12e0640 1 -- 172.21.15.105:0/1341578325 wait complete. 2024-03-29T11:17:43.867 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:43.890+0000 7f46b12e0640 1 librados: shutdown 2024-03-29T11:17:43.883 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:50.690 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.866+0000 7fc41e354640 10 monclient: get_monmap_and_config 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.866+0000 7fc41e354640 10 monclient: build_initial_monmap 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.866+0000 7fc41e354640 10 monclient: monmap: 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:50.867218+0000 2024-03-29T11:17:50.843 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:50.867218+0000 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:50.844 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.845 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.868+0000 7fc41e354640 1 Processor -- start 2024-03-29T11:17:50.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.869+0000 7fc41e354640 1 -- start start 2024-03-29T11:17:50.846 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.869+0000 7fc41e354640 10 monclient: init 2024-03-29T11:17:50.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.870+0000 7fc41e354640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:50.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.870+0000 7fc41e354640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:50.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.870+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-b con 0x7fc4181457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:50.847 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.870+0000 7fc417fff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fc4181457f0 0x7fc41813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:59932/0 (socket says 172.21.15.105:59932) 2024-03-29T11:17:50.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.874+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-a con 0x7fc41813f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:50.851 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.874+0000 7fc417fff640 1 -- 172.21.15.105:0/264263507 learned_addr learned my addr 172.21.15.105:0/264263507 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:50.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.874+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-c con 0x7fc4180a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:50.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.875+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180aaef0 con 0x7fc41813f650 2024-03-29T11:17:50.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.875+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418145bd0 con 0x7fc4180a9d70 2024-03-29T11:17:50.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.875+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418145eb0 con 0x7fc4181457f0 2024-03-29T11:17:50.854 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.875+0000 7fc41e354640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.875+0000 7fc41e354640 10 monclient(hunting): authenticate will time out at 1345.151713s 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548186084 0 0) 0x7fc4180aaef0 con 0x7fc41813f650 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc404003520 con 0x7fc41813f650 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819106784 0 0) 0x7fc418145eb0 con 0x7fc4181457f0 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4180aaef0 con 0x7fc4181457f0 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 667126929 0 0) 0x7fc404003520 con 0x7fc41813f650 2024-03-29T11:17:50.855 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 >> v1:172.21.15.105:6790/0 conn(0x7fc4181457f0 legacy=0x7fc41813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.856 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.876+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 >> v1:172.21.15.110:6789/0 conn(0x7fc4180a9d70 legacy=0x7fc41814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 1 monclient: found mon.noname-a 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418146340 con 0x7fc41813f650 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:50.859 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc408001690 con 0x7fc41813f650 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: dump: 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:50.860 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4080017f0 con 0x7fc41813f650 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 1 -- 172.21.15.105:0/264263507 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc408001c00 con 0x7fc41813f650 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc4177fe640 10 monclient: dump: 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:50.861 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc41e354640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc41e354640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.878+0000 7fc41e354640 4 set_mon_vals no callback set 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.879+0000 7fc41e354640 10 monclient: shutdown 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.879+0000 7fc41e354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.879+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 >> v1:172.21.15.105:6789/0 conn(0x7fc41813f650 legacy=0x7fc4180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.880+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 shutdown_connections 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.880+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 >> 172.21.15.105:0/264263507 conn(0x7fc418066a00 msgr2=0x7fc418066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.880+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 shutdown_connections 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.880+0000 7fc41e354640 1 -- 172.21.15.105:0/264263507 wait complete. 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.881+0000 7fc41e354640 10 monclient: build_initial_monmap 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.881+0000 7fc41e354640 10 monclient: monmap: 2024-03-29T11:17:50.862 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.881+0000 7fc41e354640 1 librados: starting msgr at 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.881+0000 7fc41e354640 1 librados: starting objecter 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.882+0000 7fc41e354640 1 Processor -- start 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.882+0000 7fc41e354640 1 -- start start 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.882+0000 7fc41e354640 1 librados: setting wanted keys 2024-03-29T11:17:50.863 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.882+0000 7fc41e354640 1 librados: calling monclient init 2024-03-29T11:17:50.864 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.882+0000 7fc41e354640 10 monclient: init 2024-03-29T11:17:50.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.883+0000 7fc41e354640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:50.869 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.883+0000 7fc41e354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.883+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-b con 0x7fc4180a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-a con 0x7fc41813f650 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 10 monclient(hunting): picked mon.noname-c con 0x7fc4181457f0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418071b30 con 0x7fc41813f650 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180ac1e0 con 0x7fc4180a9d70 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180ac4c0 con 0x7fc4181457f0 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.884+0000 7fc41e354640 10 monclient(hunting): authenticate will time out at 1345.160351s 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.885+0000 7fc41c8f1640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fc4181457f0 0x7fc4181e6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:35490/0 (socket says 172.21.15.105:35490) 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.885+0000 7fc41c8f1640 1 -- 172.21.15.105:0/1278867535 learned_addr learned my addr 172.21.15.105:0/1278867535 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 client.?.objecter ms_handle_connect 0x7fc4181457f0 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:50.870 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717420687 0 0) 0x7fc4180ac4c0 con 0x7fc4181457f0 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f8004eb0 con 0x7fc4181457f0 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 client.?.objecter ms_handle_connect 0x7fc4180a9d70 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.886+0000 7fc4157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528639461 0 0) 0x7fc3f8004eb0 con 0x7fc4181457f0 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4180ac4c0 con 0x7fc4181457f0 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276631934 0 0) 0x7fc4180ac1e0 con 0x7fc4180a9d70 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f8004eb0 con 0x7fc4180a9d70 2024-03-29T11:17:50.871 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc410001b70 con 0x7fc4181457f0 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): dump: 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:50.872 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2455186663 0 0) 0x7fc4180ac4c0 con 0x7fc4181457f0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 >> v1:172.21.15.110:6789/0 conn(0x7fc4180a9d70 legacy=0x7fc41806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 >> v1:172.21.15.105:6789/0 conn(0x7fc41813f650 legacy=0x7fc4181df460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 monclient: found mon.c 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418071dd0 con 0x7fc4181457f0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc4157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 10 monclient: _renew_subs 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4181ded50 con 0x7fc4181457f0 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:50.873 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 10 monclient: _renew_subs 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41813b140 con 0x7fc4181457f0 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.887+0000 7fc41e354640 1 librados: init done 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.888+0000 7fc41e354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.888+0000 7fc41e354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.888+0000 7fc41e354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.888+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc418075f10 con 0x7fc4181457f0 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.889+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc410001cd0 con 0x7fc4181457f0 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.889+0000 7fc4157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:50.874 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.889+0000 7fc416ffd640 4 set_mon_vals no callback set 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.892+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fc410004e20 con 0x7fc4181457f0 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.892+0000 7fc4157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.892+0000 7fc4157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.892+0000 7fc4157fa640 10 monclient: dump: 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:50.876 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.892+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fc41001adf0 con 0x7fc4181457f0 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7fc410084d10 con 0x7fc4181457f0 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc4157fa640 10 client.4640.objecter ms_dispatch 0x7fc418000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc4157fa640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc4157fa640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc4157fa640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.893+0000 7fc417fff640 10 monclient: get_auth_request con 0x7fc3f806e030 auth_method 0 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.894+0000 7fc4157fa640 10 client.4640.objecter ms_handle_connect 0x7fc3f806e030 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.897+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc410050730 con 0x7fc4181457f0 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.897+0000 7fc4157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.877 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:50.897+0000 7fc4157fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:51.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.219+0000 7fc41e354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.196 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.219+0000 7fc41e354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.219+0000 7fc41e354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:17:51.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.219+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc418071f60 con 0x7fc4181457f0 2024-03-29T11:17:51.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.220+0000 7fc4157fa640 1 -- 172.21.15.105:0/1278867535 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7fc410052660 con 0x7fc4181457f0 2024-03-29T11:17:51.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.220+0000 7fc4157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:51.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.220+0000 7fc4157fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:51.198 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:51.198 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.223+0000 7fc41e354640 10 librados: watch_flush enter 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 10 librados: watch_flush exit 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 >> v1:172.21.15.110:6816/1087970004 conn(0x7fc3f806e030 legacy=0x7fc3f80704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.201 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 10 monclient: shutdown 2024-03-29T11:17:51.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:51.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 >> v1:172.21.15.105:6790/0 conn(0x7fc4181457f0 legacy=0x7fc4181e6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 shutdown_connections 2024-03-29T11:17:51.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.224+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 >> 172.21.15.105:0/1278867535 conn(0x7fc418066a00 msgr2=0x7fc4180b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:51.202 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.225+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 shutdown_connections 2024-03-29T11:17:51.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.225+0000 7fc41e354640 1 -- 172.21.15.105:0/1278867535 wait complete. 2024-03-29T11:17:51.203 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:51.225+0000 7fc41e354640 1 librados: shutdown 2024-03-29T11:17:51.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:58.024 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:58.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.216+0000 7f213f0fb640 10 monclient: get_monmap_and_config 2024-03-29T11:17:58.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.216+0000 7f213f0fb640 10 monclient: build_initial_monmap 2024-03-29T11:17:58.193 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.216+0000 7f213f0fb640 10 monclient: monmap: 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:17:58.217214+0000 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:17:58.217214+0000 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:17:58.194 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.197 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.220+0000 7f213f0fb640 1 Processor -- start 2024-03-29T11:17:58.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.220+0000 7f213f0fb640 1 -- start start 2024-03-29T11:17:58.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.220+0000 7f213f0fb640 10 monclient: init 2024-03-29T11:17:58.198 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f21381422b0 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2138142690 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f21380b17d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.221+0000 7f213f0fb640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138142c80 con 0x7f21381422b0 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.222+0000 7f213f0fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380aae40 con 0x7f2138142690 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.222+0000 7f213f0fb640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213813fbd0 con 0x7f21380b17d0 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.222+0000 7f213f0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:58.199 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.222+0000 7f213f0fb640 10 monclient(hunting): authenticate will time out at 1352.497999s 2024-03-29T11:17:58.204 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.226+0000 7f213ce97640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f21381422b0 0x7f213813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:58293/0 (socket says 172.21.15.105:58293) 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.226+0000 7f213ce97640 1 -- 172.21.15.105:0/3395616852 learned_addr learned my addr 172.21.15.105:0/3395616852 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788025031 0 0) 0x7f21380aae40 con 0x7f2138142690 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:17:58.205 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.227+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2120003520 con 0x7f2138142690 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106290232 0 0) 0x7f2120003520 con 0x7f2138142690 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 >> v1:172.21.15.105:6790/0 conn(0x7f21380b17d0 legacy=0x7f21380b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 >> v1:172.21.15.105:6789/0 conn(0x7f21381422b0 legacy=0x7f213813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:58.206 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138151940 con 0x7f2138142690 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f213f0fb640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f213f0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2130001a10 con 0x7f2138142690 2024-03-29T11:17:58.210 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f21377fe640 10 monclient: dump: 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:58.211 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.228+0000 7f213f0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 <== mon.1 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2130001c30 con 0x7f2138142690 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f213f0fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f213f0fb640 4 set_mon_vals no callback set 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 1 -- 172.21.15.105:0/3395616852 <== mon.1 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2130002000 con 0x7f2138142690 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f21377fe640 10 monclient: dump: 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.212 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f213f0fb640 10 monclient: shutdown 2024-03-29T11:17:58.213 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f213f0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.229+0000 7f213f0fb640 1 -- 172.21.15.105:0/3395616852 >> v1:172.21.15.110:6789/0 conn(0x7f2138142690 legacy=0x7f21380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.230+0000 7f213f0fb640 1 -- 172.21.15.105:0/3395616852 shutdown_connections 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.230+0000 7f213f0fb640 1 -- 172.21.15.105:0/3395616852 >> 172.21.15.105:0/3395616852 conn(0x7f2138066a00 msgr2=0x7f2138066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.230+0000 7f213f0fb640 1 -- 172.21.15.105:0/3395616852 shutdown_connections 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.230+0000 7f213f0fb640 1 -- 172.21.15.105:0/3395616852 wait complete. 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 10 monclient: build_initial_monmap 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 10 monclient: monmap: 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.214 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 librados: starting msgr at 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 librados: starting objecter 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 Processor -- start 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 -- start start 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 librados: setting wanted keys 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 1 librados: calling monclient init 2024-03-29T11:17:58.215 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.232+0000 7f213f0fb640 10 monclient: init 2024-03-29T11:17:58.216 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f21380b17d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f21381422b0 addr v1:172.21.15.105:6790/0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2138142690 addr v1:172.21.15.105:6789/0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380ac730 con 0x7f2138142690 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380b81b0 con 0x7f21380b17d0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380b8490 con 0x7f21381422b0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.234+0000 7f213f0fb640 10 monclient(hunting): authenticate will time out at 1352.510126s 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.237+0000 7f213ce97640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f21380b17d0 0x7f21380b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:44469/0 (socket says 172.21.15.105:44469) 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.237+0000 7f213ce97640 1 -- 172.21.15.105:0/369935477 learned_addr learned my addr 172.21.15.105:0/369935477 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.238+0000 7f21357fa640 10 client.?.objecter ms_handle_connect 0x7f21380b17d0 2024-03-29T11:17:58.217 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.238+0000 7f21357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.238+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822598573 0 0) 0x7f21380b81b0 con 0x7f21380b17d0 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124004eb0 con 0x7f21380b17d0 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378392863 0 0) 0x7f2124004eb0 con 0x7f21380b17d0 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2124005300 con 0x7f21380b17d0 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f212c001930 con 0x7f21380b17d0 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): dump: 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:58.218 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.219 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.239+0000 7f21357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:58.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.243+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 733269717 0 0) 0x7f2124005300 con 0x7f21380b17d0 2024-03-29T11:17:58.220 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.243+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 >> v1:172.21.15.105:6790/0 conn(0x7f21381422b0 legacy=0x7f213812e860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 >> v1:172.21.15.105:6789/0 conn(0x7f2138142690 legacy=0x7f213812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 1 monclient: found mon.b 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.222 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213812f680 con 0x7f21380b17d0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 client.?.objecter ms_handle_connect 0x7f2138142690 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 10 monclient: _renew_subs 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21380ac0c0 con 0x7f21380b17d0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 10 monclient: _renew_subs 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21380acc60 con 0x7f21380b17d0 2024-03-29T11:17:58.223 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f213f0fb640 1 librados: init done 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f212c0047e0 con 0x7f21380b17d0 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f212c001930 con 0x7f21380b17d0 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f21357fa640 10 monclient: dump: 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:17:58.224 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.245+0000 7f2136ffd640 4 set_mon_vals no callback set 2024-03-29T11:17:58.225 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.246+0000 7f213f0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.247+0000 7f213f0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.247+0000 7f213f0fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.247+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2118005080 con 0x7f21380b17d0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f212c004950 con 0x7f21380b17d0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f2137fff640 10 monclient: get_auth_request con 0x7f212406df70 auth_method 0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f212c054630 con 0x7f21380b17d0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f21357fa640 10 client.4552.objecter ms_dispatch 0x7f2138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f21357fa640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.248+0000 7f21357fa640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:58.226 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.249+0000 7f21357fa640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:17:58.229 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.250+0000 7f21357fa640 10 client.4552.objecter ms_handle_connect 0x7f212406df70 2024-03-29T11:17:58.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.255+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f212c0508d0 con 0x7f21380b17d0 2024-03-29T11:17:58.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.255+0000 7f21357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:58.232 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.255+0000 7f21357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:58.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.611+0000 7f213f0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.612+0000 7f213f0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.612+0000 7f213f0fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:17:58.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.612+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2118005330 con 0x7f21380b17d0 2024-03-29T11:17:58.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.615+0000 7f21357fa640 1 -- 172.21.15.105:0/369935477 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f212c0508d0 con 0x7f21380b17d0 2024-03-29T11:17:58.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.615+0000 7f21357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.615+0000 7f21357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:58.593 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:17:58.593 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:58.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.619+0000 7f213f0fb640 10 librados: watch_flush enter 2024-03-29T11:17:58.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 10 librados: watch_flush exit 2024-03-29T11:17:58.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:17:58.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:17:58.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 >> v1:172.21.15.110:6816/1087970004 conn(0x7f212406df70 legacy=0x7f2124070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 10 monclient: shutdown 2024-03-29T11:17:58.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:58.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.622+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 >> v1:172.21.15.110:6789/0 conn(0x7f21380b17d0 legacy=0x7f21380b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.623+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 shutdown_connections 2024-03-29T11:17:58.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.623+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 >> 172.21.15.105:0/369935477 conn(0x7f2138066a00 msgr2=0x7f213813e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:58.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.624+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 shutdown_connections 2024-03-29T11:17:58.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.624+0000 7f213f0fb640 1 -- 172.21.15.105:0/369935477 wait complete. 2024-03-29T11:17:58.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:17:58.624+0000 7f213f0fb640 1 librados: shutdown 2024-03-29T11:17:58.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:05.428 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.598+0000 7f8d8359e640 10 monclient: get_monmap_and_config 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.598+0000 7f8d8359e640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.598+0000 7f8d8359e640 10 monclient: monmap: 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:05.599372+0000 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:05.599372+0000 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.577 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.600+0000 7f8d8359e640 1 Processor -- start 2024-03-29T11:18:05.578 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.600+0000 7f8d8359e640 1 -- start start 2024-03-29T11:18:05.578 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.600+0000 7f8d8359e640 10 monclient: init 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7c02cc50 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7c02d3a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7c0c6290 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0c56f0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c02e080 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c02e360 con 0x7f8d7c02cc50 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8359e640 10 monclient(hunting): authenticate will time out at 1359.877151s 2024-03-29T11:18:05.580 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d82d9d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f8d7c0c6290 0x7f8d7c0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:49712/0 (socket says 172.21.15.105:49712) 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d82d9d640 1 -- 172.21.15.105:0/1945877511 learned_addr learned my addr 172.21.15.105:0/1945877511 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710501208 0 0) 0x7f8d7c0c56f0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.601+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c003520 con 0x7f8d7c0c6290 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.602+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007018838 0 0) 0x7f8d6c003520 con 0x7f8d7c0c6290 2024-03-29T11:18:05.581 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.602+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 >> v1:172.21.15.105:6790/0 conn(0x7f8d7c02cc50 legacy=0x7f8d7c0c2ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 >> v1:172.21.15.110:6789/0 conn(0x7f8d7c02d3a0 legacy=0x7f8d7c0c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 1 monclient: found mon.noname-a 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:18:05.582 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0c8c10 con 0x7f8d7c0c6290 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8359e640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d8404f730 con 0x7f8d7c0c6290 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: dump: 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.583 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d840008d0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:05.584 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.605+0000 7f8d8359e640 4 set_mon_vals no callback set 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.606+0000 7f8d8159a640 1 -- 172.21.15.105:0/1945877511 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d840758b0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.606+0000 7f8d8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.606+0000 7f8d8159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.606+0000 7f8d8159a640 10 monclient: dump: 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.585 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.606+0000 7f8d8359e640 10 monclient: shutdown 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.607+0000 7f8d8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.607+0000 7f8d8359e640 1 -- 172.21.15.105:0/1945877511 >> v1:172.21.15.105:6789/0 conn(0x7f8d7c0c6290 legacy=0x7f8d7c0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.607+0000 7f8d8359e640 1 -- 172.21.15.105:0/1945877511 shutdown_connections 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.607+0000 7f8d8359e640 1 -- 172.21.15.105:0/1945877511 >> 172.21.15.105:0/1945877511 conn(0x7f8d7c0058f0 msgr2=0x7f8d7c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.608+0000 7f8d8359e640 1 -- 172.21.15.105:0/1945877511 shutdown_connections 2024-03-29T11:18:05.586 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.608+0000 7f8d8359e640 1 -- 172.21.15.105:0/1945877511 wait complete. 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 10 monclient: monmap: 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:05.587 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 librados: starting msgr at 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 librados: starting objecter 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 Processor -- start 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 -- start start 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 librados: setting wanted keys 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 1 librados: calling monclient init 2024-03-29T11:18:05.588 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.609+0000 7f8d8359e640 10 monclient: init 2024-03-29T11:18:05.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:05.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7c02cc50 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:05.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7c02d3a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:05.589 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7c0c6290 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:05.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c150630 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c150910 con 0x7f8d7c02cc50 2024-03-29T11:18:05.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c160f00 con 0x7f8d7c0c6290 2024-03-29T11:18:05.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.590 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.610+0000 7f8d8359e640 10 monclient(hunting): authenticate will time out at 1359.886690s 2024-03-29T11:18:05.593 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.614+0000 7f8d82d9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f8d7c0c6290 0x7f8d7c16c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:44533/0 (socket says 172.21.15.105:44533) 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.614+0000 7f8d82d9d640 1 -- 172.21.15.105:0/2924844940 learned_addr learned my addr 172.21.15.105:0/2924844940 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.616+0000 7f8d72ffd640 10 client.?.objecter ms_handle_connect 0x7f8d7c0c6290 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 client.?.objecter ms_handle_connect 0x7f8d7c02d3a0 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 client.?.objecter ms_handle_connect 0x7f8d7c02cc50 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012249740 0 0) 0x7f8d7c160f00 con 0x7f8d7c0c6290 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.594 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64004eb0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323497076 0 0) 0x7f8d7c150910 con 0x7f8d7c02cc50 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c160f00 con 0x7f8d7c02cc50 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650362132 0 0) 0x7f8d7c150630 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:18:05.595 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c150910 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446590033 0 0) 0x7f8d64004eb0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.597 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7c150630 con 0x7f8d7c0c6290 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d8404f400 con 0x7f8d7c0c6290 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.598 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3829380570 0 0) 0x7f8d7c160f00 con 0x7f8d7c02cc50 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d64004eb0 con 0x7f8d7c02cc50 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3296504309 0 0) 0x7f8d7c150910 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d64006d70 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d78001830 con 0x7f8d7c02d3a0 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.599 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d74001680 con 0x7f8d7c02cc50 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.600 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 327882047 0 0) 0x7f8d7c150630 con 0x7f8d7c0c6290 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 >> v1:172.21.15.110:6789/0 conn(0x7f8d7c02cc50 legacy=0x7f8d7c161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 >> v1:172.21.15.105:6789/0 conn(0x7f8d7c02d3a0 legacy=0x7f8d7c0c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 monclient: found mon.c 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:05.601 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c02fe60 con 0x7f8d7c0c6290 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.617+0000 7f8d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d8359e640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d8359e640 10 monclient: _renew_subs 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c161b30 con 0x7f8d7c0c6290 2024-03-29T11:18:05.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d8404f730 con 0x7f8d7c0c6290 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8d84067d90 con 0x7f8d7c0c6290 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d72ffd640 10 monclient: dump: 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.618+0000 7f8d80d99640 4 set_mon_vals no callback set 2024-03-29T11:18:05.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.619+0000 7f8d8359e640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.619+0000 7f8d8359e640 10 monclient: _renew_subs 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.619+0000 7f8d8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.619+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c032b10 con 0x7f8d7c0c6290 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.619+0000 7f8d8359e640 1 librados: init done 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8d84067ef0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8d840cf240 con 0x7f8d7c0c6290 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 10 client.4643.objecter ms_dispatch 0x7f8d84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.620+0000 7f8d72ffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.621+0000 7f8d8259c640 10 monclient: get_auth_request con 0x7f8d6406e520 auth_method 0 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.622+0000 7f8d72ffd640 10 client.4643.objecter ms_handle_connect 0x7f8d6406e520 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.622+0000 7f8d8359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.622+0000 7f8d8359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.606 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.622+0000 7f8d8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:05.607 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.622+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c02fff0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.632+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d840cad50 con 0x7f8d7c0c6290 2024-03-29T11:18:05.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.632+0000 7f8d72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.609 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.632+0000 7f8d72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:05.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.971+0000 7f8d8359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.971+0000 7f8d8359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.971+0000 7f8d8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:05.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.971+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d7c004cd0 con 0x7f8d7c0c6290 2024-03-29T11:18:05.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.975+0000 7f8d72ffd640 1 -- 172.21.15.105:0/2924844940 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8d840ccc80 con 0x7f8d7c0c6290 2024-03-29T11:18:05.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.975+0000 7f8d72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.975+0000 7f8d72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:05.955 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:05.955 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:05.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.981+0000 7f8d8359e640 10 librados: watch_flush enter 2024-03-29T11:18:05.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 10 librados: watch_flush exit 2024-03-29T11:18:05.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:18:05.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 >> v1:172.21.15.110:6816/1087970004 conn(0x7f8d6406e520 legacy=0x7f8d640709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 10 monclient: shutdown 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 >> v1:172.21.15.105:6790/0 conn(0x7f8d7c0c6290 legacy=0x7f8d7c16c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 shutdown_connections 2024-03-29T11:18:05.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.982+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 >> 172.21.15.105:0/2924844940 conn(0x7f8d7c0058f0 msgr2=0x7f8d7c038460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:05.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.984+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 shutdown_connections 2024-03-29T11:18:05.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.984+0000 7f8d8359e640 1 -- 172.21.15.105:0/2924844940 wait complete. 2024-03-29T11:18:05.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:05.984+0000 7f8d8359e640 1 librados: shutdown 2024-03-29T11:18:05.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:12.787 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:12.927 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.950+0000 7f03e8c02640 10 monclient: get_monmap_and_config 2024-03-29T11:18:12.927 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.950+0000 7f03e8c02640 10 monclient: build_initial_monmap 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.950+0000 7f03e8c02640 10 monclient: monmap: 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:12.951261+0000 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:12.951261+0000 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:12.928 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.929 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.952+0000 7f03e8c02640 1 Processor -- start 2024-03-29T11:18:12.929 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.952+0000 7f03e8c02640 1 -- start start 2024-03-29T11:18:12.929 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.952+0000 7f03e8c02640 10 monclient: init 2024-03-29T11:18:12.930 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:12.930 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:12.930 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-b con 0x7f03dc006800 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:12.930 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-a con 0x7f03dc006be0 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-c con 0x7f03dc031b60 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc029370 con 0x7f03dc006be0 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0295e0 con 0x7f03dc031b60 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc031360 con 0x7f03dc006800 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e8c02640 10 monclient(hunting): authenticate will time out at 1367.229295s 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e37fe640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f03dc006800 0x7f03dc0cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:41912/0 (socket says 172.21.15.105:41912) 2024-03-29T11:18:12.931 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e37fe640 1 -- 172.21.15.105:0/2462889609 learned_addr learned my addr 172.21.15.105:0/2462889609 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830581225 0 0) 0x7f03dc031360 con 0x7f03dc006800 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.953+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d4003520 con 0x7f03dc006800 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715449679 0 0) 0x7f03dc029370 con 0x7f03dc006be0 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:18:12.933 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc031360 con 0x7f03dc006be0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110487463 0 0) 0x7f03dc031360 con 0x7f03dc006be0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 >> v1:172.21.15.105:6790/0 conn(0x7f03dc006800 legacy=0x7f03dc0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 >> v1:172.21.15.110:6789/0 conn(0x7f03dc031b60 legacy=0x7f03dc0d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 monclient: found mon.noname-a 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc031640 con 0x7f03dc006be0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e8c02640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f03d0001670 con 0x7f03dc006be0 2024-03-29T11:18:12.934 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: dump: 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e8c02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03d00017d0 con 0x7f03dc006be0 2024-03-29T11:18:12.935 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 1 -- 172.21.15.105:0/2462889609 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f03d0001b60 con 0x7f03dc006be0 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e27fc640 10 monclient: dump: 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.936 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e8c02640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.954+0000 7f03e8c02640 4 set_mon_vals no callback set 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 10 monclient: shutdown 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 1 -- 172.21.15.105:0/2462889609 >> v1:172.21.15.105:6789/0 conn(0x7f03dc006be0 legacy=0x7f03dc0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 1 -- 172.21.15.105:0/2462889609 shutdown_connections 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 1 -- 172.21.15.105:0/2462889609 >> 172.21.15.105:0/2462889609 conn(0x7f03dc0058f0 msgr2=0x7f03dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 1 -- 172.21.15.105:0/2462889609 shutdown_connections 2024-03-29T11:18:12.937 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.955+0000 7f03e8c02640 1 -- 172.21.15.105:0/2462889609 wait complete. 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.956+0000 7f03e8c02640 10 monclient: build_initial_monmap 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.956+0000 7f03e8c02640 10 monclient: monmap: 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:12.945 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.956+0000 7f03e8c02640 1 librados: starting msgr at 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.956+0000 7f03e8c02640 1 librados: starting objecter 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.957+0000 7f03e8c02640 1 Processor -- start 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.957+0000 7f03e8c02640 1 -- start start 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.957+0000 7f03e8c02640 1 librados: setting wanted keys 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.957+0000 7f03e8c02640 1 librados: calling monclient init 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.957+0000 7f03e8c02640 10 monclient: init 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-a con 0x7f03dc006800 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-c con 0x7f03dc031b60 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient(hunting): picked mon.noname-b con 0x7f03dc0ca480 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0bf760 con 0x7f03dc006800 2024-03-29T11:18:12.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0a21d0 con 0x7f03dc0ca480 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0ca9b0 con 0x7f03dc031b60 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e8c02640 10 monclient(hunting): authenticate will time out at 1367.234414s 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e37fe640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f03dc006800 0x7f03dc15c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:37118/0 (socket says 172.21.15.105:37118) 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03e37fe640 1 -- 172.21.15.105:0/3459519366 learned_addr learned my addr 172.21.15.105:0/3459519366 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03cffff640 10 client.?.objecter ms_handle_connect 0x7f03dc031b60 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03cffff640 10 client.?.objecter ms_handle_connect 0x7f03dc006800 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.958+0000 7f03cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228192635 0 0) 0x7f03dc0ca9b0 con 0x7f03dc031b60 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:18:12.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.959+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c8004eb0 con 0x7f03dc031b60 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961107998 0 0) 0x7f03dc0bf760 con 0x7f03dc006800 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc0ca9b0 con 0x7f03dc006800 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773353793 0 0) 0x7f03c8004eb0 con 0x7f03dc031b60 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03dc0bf760 con 0x7f03dc031b60 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f03d0001ae0 con 0x7f03dc031b60 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): dump: 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.948 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.960+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154239815 0 0) 0x7f03dc0ca9b0 con 0x7f03dc006800 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03c8004eb0 con 0x7f03dc006800 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f03d80018d0 con 0x7f03dc006800 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 10 monclient(hunting): dump: 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:12.949 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2300714163 0 0) 0x7f03c8004eb0 con 0x7f03dc006800 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 >> v1:172.21.15.105:6790/0 conn(0x7f03dc031b60 legacy=0x7f03dc0be940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 >> v1:172.21.15.110:6789/0 conn(0x7f03dc0ca480 legacy=0x7f03dc0bf050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.961+0000 7f03cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 1 monclient: found mon.a 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:12.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0cab40 con 0x7f03dc006800 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03d80047e0 con 0x7f03dc006800 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 10 monclient: _renew_subs 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc0bf980 con 0x7f03dc006800 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:12.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 10 monclient: _renew_subs 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc0a0ce0 con 0x7f03dc006800 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.962+0000 7f03e8c02640 1 librados: init done 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.963+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f03d8004bf0 con 0x7f03dc006800 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.963+0000 7f03cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.963+0000 7f03cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.963+0000 7f03cffff640 10 monclient: dump: 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:12.952 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.964+0000 7f03e8c02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.964+0000 7f03e8c02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.964+0000 7f03e8c02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.964+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc004cd0 con 0x7f03dc006800 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.965+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f03d8004d50 con 0x7f03dc006800 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f03d80849b0 con 0x7f03dc006800 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03e2ffd640 10 monclient: get_auth_request con 0x7f03c806dec0 auth_method 0 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03cffff640 10 client.4572.objecter ms_dispatch 0x7f03e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03cffff640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03cffff640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.966+0000 7f03cffff640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:18:12.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.967+0000 7f03cffff640 10 client.4572.objecter ms_handle_connect 0x7f03c806dec0 2024-03-29T11:18:12.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.973+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03d8050290 con 0x7f03dc006800 2024-03-29T11:18:12.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.974+0000 7f03cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:12.974+0000 7f03cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:13.269 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.291+0000 7f03e8c02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.269 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.291+0000 7f03e8c02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.269 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.291+0000 7f03e8c02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:13.269 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.291+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03dc0086f0 con 0x7f03dc006800 2024-03-29T11:18:13.269 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.292+0000 7f03cffff640 1 -- 172.21.15.105:0/3459519366 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f03d80521c0 con 0x7f03dc006800 2024-03-29T11:18:13.270 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.292+0000 7f03cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:13.270 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.292+0000 7f03cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:13.270 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:13.270 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 10 librados: watch_flush enter 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 10 librados: watch_flush exit 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 >> v1:172.21.15.110:6816/1087970004 conn(0x7f03c806dec0 legacy=0x7f03c8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 10 monclient: shutdown 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 >> v1:172.21.15.105:6789/0 conn(0x7f03dc006800 legacy=0x7f03dc15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 shutdown_connections 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 >> 172.21.15.105:0/3459519366 conn(0x7f03dc0058f0 msgr2=0x7f03dc02a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 shutdown_connections 2024-03-29T11:18:13.274 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.296+0000 7f03e8c02640 1 -- 172.21.15.105:0/3459519366 wait complete. 2024-03-29T11:18:13.275 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:13.297+0000 7f03e8c02640 1 librados: shutdown 2024-03-29T11:18:13.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:20.094 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:20.239 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.262+0000 7f8a1b700640 10 monclient: get_monmap_and_config 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.262+0000 7f8a1b700640 10 monclient: build_initial_monmap 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.262+0000 7f8a1b700640 10 monclient: monmap: 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:20.263314+0000 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:20.263314+0000 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:20.240 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.241 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:20.241 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:20.241 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:20.241 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.263+0000 7f8a1b700640 1 Processor -- start 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.263+0000 7f8a1b700640 1 -- start start 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.263+0000 7f8a1b700640 10 monclient: init 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-a con 0x7f8a141457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-c con 0x7f8a140a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140aaef0 con 0x7f8a141457f0 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a14145bd0 con 0x7f8a140a9d70 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a14145eb0 con 0x7f8a1413f650 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:20.250 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1b700640 10 monclient(hunting): authenticate will time out at 1374.540585s 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1949c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f8a141457f0 0x7f8a1413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:43062/0 (socket says 172.21.15.105:43062) 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.264+0000 7f8a1949c640 1 -- 172.21.15.105:0/3419597279 learned_addr learned my addr 172.21.15.105:0/3419597279 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018151967 0 0) 0x7f8a140aaef0 con 0x7f8a141457f0 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc003520 con 0x7f8a141457f0 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 383950971 0 0) 0x7f89fc003520 con 0x7f8a141457f0 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 >> v1:172.21.15.105:6790/0 conn(0x7f8a1413f650 legacy=0x7f8a140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 >> v1:172.21.15.110:6789/0 conn(0x7f8a140a9d70 legacy=0x7f8a1414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 monclient: found mon.noname-a 2024-03-29T11:18:20.251 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a14146340 con 0x7f8a141457f0 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a0c001690 con 0x7f8a141457f0 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: dump: 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.252 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a1b700640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a1b700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a0c0018f0 con 0x7f8a141457f0 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 1 -- 172.21.15.105:0/3419597279 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a0c001d40 con 0x7f8a141457f0 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a13fff640 10 monclient: dump: 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:20.253 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a1b700640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.265+0000 7f8a1b700640 4 set_mon_vals no callback set 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 10 monclient: shutdown 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:20.254 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 1 -- 172.21.15.105:0/3419597279 >> v1:172.21.15.105:6789/0 conn(0x7f8a141457f0 legacy=0x7f8a1413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 1 -- 172.21.15.105:0/3419597279 shutdown_connections 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 1 -- 172.21.15.105:0/3419597279 >> 172.21.15.105:0/3419597279 conn(0x7f8a14066a00 msgr2=0x7f8a14066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 1 -- 172.21.15.105:0/3419597279 shutdown_connections 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.266+0000 7f8a1b700640 1 -- 172.21.15.105:0/3419597279 wait complete. 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.267+0000 7f8a1b700640 10 monclient: build_initial_monmap 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.267+0000 7f8a1b700640 10 monclient: monmap: 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:20.255 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 librados: starting msgr at 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 librados: starting objecter 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 Processor -- start 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 -- start start 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 librados: setting wanted keys 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 1 librados: calling monclient init 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.268+0000 7f8a1b700640 10 monclient: init 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-b con 0x7f8a140a9d70 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1413f650 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:20.256 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient(hunting): picked mon.noname-a con 0x7f8a141457f0 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140ad7e0 con 0x7f8a141457f0 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140af2b0 con 0x7f8a140a9d70 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140ad110 con 0x7f8a1413f650 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a1b700640 10 monclient(hunting): authenticate will time out at 1374.545407s 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a18c9b640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f8a1413f650 0x7f8a140adc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55528/0 (socket says 172.21.15.105:55528) 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a18c9b640 1 -- 172.21.15.105:0/3621449372 learned_addr learned my addr 172.21.15.105:0/3621449372 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 client.?.objecter ms_handle_connect 0x7f8a1413f650 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540183323 0 0) 0x7f8a140ad110 con 0x7f8a1413f650 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.257 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.269+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00004eb0 con 0x7f8a1413f650 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 client.?.objecter ms_handle_connect 0x7f8a141457f0 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183728734 0 0) 0x7f8a140ad7e0 con 0x7f8a141457f0 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a140ad110 con 0x7f8a141457f0 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1389053514 0 0) 0x7f8a00004eb0 con 0x7f8a1413f650 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a140ad7e0 con 0x7f8a1413f650 2024-03-29T11:18:20.258 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a08001990 con 0x7f8a1413f650 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:20.259 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2916638946 0 0) 0x7f8a140ad7e0 con 0x7f8a1413f650 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 >> v1:172.21.15.110:6789/0 conn(0x7f8a140a9d70 legacy=0x7f8a141d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.270+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 >> v1:172.21.15.105:6789/0 conn(0x7f8a141457f0 legacy=0x7f8a140b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 1 monclient: found mon.c 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a140ad2a0 con 0x7f8a1413f650 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 10 monclient: _renew_subs 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a14141fe0 con 0x7f8a1413f650 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:20.260 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 10 monclient: _renew_subs 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a08001af0 con 0x7f8a1413f650 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a140b1120 con 0x7f8a1413f650 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8a08003a60 con 0x7f8a1413f650 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.272+0000 7f8a11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.271+0000 7f8a137fe640 4 set_mon_vals no callback set 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.272+0000 7f8a11ffb640 10 monclient: dump: 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:20.261 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:20.262 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:20.262 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:20.262 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.272+0000 7f8a1b700640 1 librados: init done 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.273+0000 7f8a1b700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.273+0000 7f8a1b700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.273+0000 7f8a1b700640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.273+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a141d7f90 con 0x7f8a1413f650 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f8a08003bc0 con 0x7f8a1413f650 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8a08084870 con 0x7f8a1413f650 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 10 client.4652.objecter ms_dispatch 0x7f8a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:20.263 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a11ffb640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.275+0000 7f8a1949c640 10 monclient: get_auth_request con 0x7f8a0006dad0 auth_method 0 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.276+0000 7f8a11ffb640 10 client.4652.objecter ms_handle_connect 0x7f8a0006dad0 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.284+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a0804ff80 con 0x7f8a1413f650 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.284+0000 7f8a11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:20.264 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.284+0000 7f8a11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:20.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.647+0000 7f8a1b700640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.647+0000 7f8a1b700640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.647+0000 7f8a1b700640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:20.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.647+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a141d7f90 con 0x7f8a1413f650 2024-03-29T11:18:20.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.649+0000 7f8a11ffb640 1 -- 172.21.15.105:0/3621449372 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f8a141d7f90 con 0x7f8a1413f650 2024-03-29T11:18:20.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.649+0000 7f8a11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.649+0000 7f8a11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:20.626 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:20.626 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:20.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 10 librados: watch_flush enter 2024-03-29T11:18:20.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 10 librados: watch_flush exit 2024-03-29T11:18:20.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:18:20.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 >> v1:172.21.15.110:6816/1087970004 conn(0x7f8a0006dad0 legacy=0x7f8a0006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 10 monclient: shutdown 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.652+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 >> v1:172.21.15.105:6790/0 conn(0x7f8a1413f650 legacy=0x7f8a140adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.653+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 shutdown_connections 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.653+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 >> 172.21.15.105:0/3621449372 conn(0x7f8a14066a00 msgr2=0x7f8a1413cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.653+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 shutdown_connections 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.653+0000 7f8a1b700640 1 -- 172.21.15.105:0/3621449372 wait complete. 2024-03-29T11:18:20.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:20.653+0000 7f8a1b700640 1 librados: shutdown 2024-03-29T11:18:20.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:27.452 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:27.599 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.622+0000 7f84ab2f1640 10 monclient: get_monmap_and_config 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.622+0000 7f84ab2f1640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.622+0000 7f84ab2f1640 10 monclient: monmap: 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:27.623785+0000 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:27.623785+0000 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:27.600 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.603 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.624+0000 7f84ab2f1640 1 Processor -- start 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.624+0000 7f84ab2f1640 1 -- start start 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.624+0000 7f84ab2f1640 10 monclient: init 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f84a40b1f60 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f84a4071eb0 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f84a4143d10 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a406ef80 con 0x7f84a40b1f60 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a406f260 con 0x7f84a4071eb0 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40b40c0 con 0x7f84a4143d10 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.604 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.625+0000 7f84ab2f1640 10 monclient(hunting): authenticate will time out at 1381.901435s 2024-03-29T11:18:27.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.628+0000 7f84a908d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f84a40b1f60 0x7f84a4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:46860/0 (socket says 172.21.15.105:46860) 2024-03-29T11:18:27.605 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.628+0000 7f84a908d640 1 -- 172.21.15.105:0/3081295062 learned_addr learned my addr 172.21.15.105:0/3081295062 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206695823 0 0) 0x7f84a40b40c0 con 0x7f84a4143d10 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8490003520 con 0x7f84a4143d10 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 875340762 0 0) 0x7f8490003520 con 0x7f84a4143d10 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 >> v1:172.21.15.110:6789/0 conn(0x7f84a4071eb0 legacy=0x7f84a4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 >> v1:172.21.15.105:6789/0 conn(0x7f84a40b1f60 legacy=0x7f84a4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.615 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 monclient: found mon.noname-b 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4155d70 con 0x7f84a4143d10 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f84a0001690 con 0x7f84a4143d10 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: dump: 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:27.616 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f84ab2f1640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f84ab2f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84a0001860 con 0x7f84a4143d10 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 1 -- 172.21.15.105:0/3081295062 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f84a0001cb0 con 0x7f84a4143d10 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f849bfff640 10 monclient: dump: 2024-03-29T11:18:27.617 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f84ab2f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.629+0000 7f84ab2f1640 4 set_mon_vals no callback set 2024-03-29T11:18:27.618 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.630+0000 7f84ab2f1640 10 monclient: shutdown 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.630+0000 7f84ab2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.630+0000 7f84ab2f1640 1 -- 172.21.15.105:0/3081295062 >> v1:172.21.15.105:6790/0 conn(0x7f84a4143d10 legacy=0x7f84a4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.630+0000 7f84ab2f1640 1 -- 172.21.15.105:0/3081295062 shutdown_connections 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.630+0000 7f84ab2f1640 1 -- 172.21.15.105:0/3081295062 >> 172.21.15.105:0/3081295062 conn(0x7f84a4066a00 msgr2=0x7f84a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.631+0000 7f84ab2f1640 1 -- 172.21.15.105:0/3081295062 shutdown_connections 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.631+0000 7f84ab2f1640 1 -- 172.21.15.105:0/3081295062 wait complete. 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.632+0000 7f84ab2f1640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.632+0000 7f84ab2f1640 10 monclient: monmap: 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:27.619 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.632+0000 7f84ab2f1640 1 librados: starting msgr at 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.632+0000 7f84ab2f1640 1 librados: starting objecter 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 Processor -- start 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 -- start start 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 librados: setting wanted keys 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 librados: calling monclient init 2024-03-29T11:18:27.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient: init 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f84a40b1f60 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f84a4143d10 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f84a414a270 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a4149de0 con 0x7f84a40b1f60 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a414b090 con 0x7f84a414a270 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a413bc00 con 0x7f84a4143d10 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.633+0000 7f84ab2f1640 10 monclient(hunting): authenticate will time out at 1381.909970s 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f84a908d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f84a40b1f60 0x7f84a4147790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:50240/0 (socket says 172.21.15.105:50240) 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f84a908d640 1 -- 172.21.15.105:0/1742119551 learned_addr learned my addr 172.21.15.105:0/1742119551 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 client.?.objecter ms_handle_connect 0x7f84a40b1f60 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558313073 0 0) 0x7f84a4149de0 con 0x7f84a40b1f60 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488004eb0 con 0x7f84a40b1f60 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.635+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268197672 0 0) 0x7f8488004eb0 con 0x7f84a40b1f60 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8488005300 con 0x7f84a40b1f60 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8494001990 con 0x7f84a40b1f60 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.622 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241856982 0 0) 0x7f8488005300 con 0x7f84a40b1f60 2024-03-29T11:18:27.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.636+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 >> v1:172.21.15.105:6790/0 conn(0x7f84a4143d10 legacy=0x7f84a413ade0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 >> v1:172.21.15.110:6789/0 conn(0x7f84a414a270 legacy=0x7f84a413b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 1 monclient: found mon.a 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a414b2e0 con 0x7f84a40b1f60 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 client.?.objecter ms_handle_connect 0x7f84a414a270 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84940047e0 con 0x7f84a40b1f60 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f8499ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.639+0000 7f849b7fe640 4 set_mon_vals no callback set 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f8494001990 con 0x7f84a40b1f60 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f8499ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f8499ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f8499ffb640 10 monclient: dump: 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:27.627 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 10 monclient: _renew_subs 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4149970 con 0x7f84a40b1f60 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 10 monclient: _renew_subs 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4148080 con 0x7f84a40b1f60 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.640+0000 7f84ab2f1640 1 librados: init done 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.641+0000 7f84ab2f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.641+0000 7f84ab2f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.641+0000 7f84ab2f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.641+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4149bb0 con 0x7f84a40b1f60 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.642+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f849401a900 con 0x7f84a40b1f60 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.643+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 652950522 0 0) 0x7f8494084490 con 0x7f84a40b1f60 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.643+0000 7f8499ffb640 10 client.4584.objecter ms_dispatch 0x7f84a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.643+0000 7f8499ffb640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.643+0000 7f8499ffb640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.643+0000 7f8499ffb640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.644+0000 7f84a888c640 10 monclient: get_auth_request con 0x7f848806e020 auth_method 0 2024-03-29T11:18:27.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.646+0000 7f8499ffb640 10 client.4584.objecter ms_handle_connect 0x7f848806e020 2024-03-29T11:18:27.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.650+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f849404fdd0 con 0x7f84a40b1f60 2024-03-29T11:18:27.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.650+0000 7f8499ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.650+0000 7f8499ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:27.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.983+0000 7f84ab2f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.983+0000 7f84ab2f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.983+0000 7f84ab2f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:27.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.983+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84a4148080 con 0x7f84a40b1f60 2024-03-29T11:18:27.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.984+0000 7f8499ffb640 1 -- 172.21.15.105:0/1742119551 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4039709281) 0x7f84a4148080 con 0x7f84a40b1f60 2024-03-29T11:18:27.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.984+0000 7f8499ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.984+0000 7f8499ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:27.962 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:27.962 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:08:31.219187+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:08:31.219187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.987+0000 7f84ab2f1640 10 librados: watch_flush enter 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.987+0000 7f84ab2f1640 10 librados: watch_flush exit 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.987+0000 7f84ab2f1640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.987+0000 7f84ab2f1640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.988+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 >> v1:172.21.15.110:6816/1087970004 conn(0x7f848806e020 legacy=0x7f84880704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.988+0000 7f84ab2f1640 10 monclient: shutdown 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.988+0000 7f84ab2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:27.965 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.988+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 >> v1:172.21.15.105:6789/0 conn(0x7f84a40b1f60 legacy=0x7f84a4147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.990+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 shutdown_connections 2024-03-29T11:18:27.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.990+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 >> 172.21.15.105:0/1742119551 conn(0x7f84a4066a00 msgr2=0x7f84a40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:27.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.990+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 shutdown_connections 2024-03-29T11:18:27.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.990+0000 7f84ab2f1640 1 -- 172.21.15.105:0/1742119551 wait complete. 2024-03-29T11:18:27.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:27.991+0000 7f84ab2f1640 1 librados: shutdown 2024-03-29T11:18:27.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:34.794 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:34.941 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.963+0000 7f30fb6c1640 10 monclient: get_monmap_and_config 2024-03-29T11:18:34.941 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.963+0000 7f30fb6c1640 10 monclient: build_initial_monmap 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.963+0000 7f30fb6c1640 10 monclient: monmap: 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:34.964583+0000 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:34.964583+0000 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.942 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.964+0000 7f30fb6c1640 1 Processor -- start 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.964+0000 7f30fb6c1640 1 -- start start 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.964+0000 7f30fb6c1640 10 monclient: init 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-a con 0x7f30f40b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-b con 0x7f30f414a700 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:34.943 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-c con 0x7f30f40ac5a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:34.944 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40aba70 con 0x7f30f40b0770 2024-03-29T11:18:34.944 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40aecb0 con 0x7f30f40ac5a0 2024-03-29T11:18:34.944 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40af150 con 0x7f30f414a700 2024-03-29T11:18:34.944 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:34.944 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.965+0000 7f30fb6c1640 10 monclient(hunting): authenticate will time out at 1389.241666s 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30f8c5c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f30f414a700 0x7f30f40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:44036/0 (socket says 172.21.15.105:44036) 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30f8c5c640 1 -- 172.21.15.105:0/837981081 learned_addr learned my addr 172.21.15.105:0/837981081 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378241524 0 0) 0x7f30f40af150 con 0x7f30f414a700 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.946 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.969+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e0003520 con 0x7f30f414a700 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896116544 0 0) 0x7f30e0003520 con 0x7f30f414a700 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 >> v1:172.21.15.110:6789/0 conn(0x7f30f40ac5a0 legacy=0x7f30f4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 >> v1:172.21.15.105:6789/0 conn(0x7f30f40b0770 legacy=0x7f30f4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 monclient: found mon.noname-b 2024-03-29T11:18:34.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40ac9d0 con 0x7f30f414a700 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f30e4001690 con 0x7f30f414a700 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.970+0000 7f30ebfff640 10 monclient: dump: 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.948 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.949 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.949 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:34.949 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:34.949 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:34.949 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e40017f0 con 0x7f30f414a700 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 1 -- 172.21.15.105:0/837981081 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f30e4001c40 con 0x7f30f414a700 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30ebfff640 10 monclient: dump: 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.950 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30fb6c1640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30fb6c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.971+0000 7f30fb6c1640 4 set_mon_vals no callback set 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.972+0000 7f30fb6c1640 10 monclient: shutdown 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.972+0000 7f30fb6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:34.951 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.972+0000 7f30fb6c1640 1 -- 172.21.15.105:0/837981081 >> v1:172.21.15.105:6790/0 conn(0x7f30f414a700 legacy=0x7f30f40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.973+0000 7f30fb6c1640 1 -- 172.21.15.105:0/837981081 shutdown_connections 2024-03-29T11:18:34.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.973+0000 7f30fb6c1640 1 -- 172.21.15.105:0/837981081 >> 172.21.15.105:0/837981081 conn(0x7f30f406b660 msgr2=0x7f30f406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:34.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.974+0000 7f30fb6c1640 1 -- 172.21.15.105:0/837981081 shutdown_connections 2024-03-29T11:18:34.952 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.974+0000 7f30fb6c1640 1 -- 172.21.15.105:0/837981081 wait complete. 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.975+0000 7f30fb6c1640 10 monclient: build_initial_monmap 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.975+0000 7f30fb6c1640 10 monclient: monmap: 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.953 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.975+0000 7f30fb6c1640 1 librados: starting msgr at 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.975+0000 7f30fb6c1640 1 librados: starting objecter 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.975+0000 7f30fb6c1640 1 Processor -- start 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 -- start start 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 librados: setting wanted keys 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 librados: calling monclient init 2024-03-29T11:18:34.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient: init 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-c con 0x7f30f40ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-a con 0x7f30f40b0770 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient(hunting): picked mon.noname-b con 0x7f30f414a700 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f407deb0 con 0x7f30f40b0770 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f41e9ba0 con 0x7f30f414a700 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f41e9d30 con 0x7f30f40ac5a0 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.976+0000 7f30fb6c1640 10 monclient(hunting): authenticate will time out at 1389.252902s 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30f8c5c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f30f40b0770 0x7f30f41ec2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:39987/0 (socket says 172.21.15.105:39987) 2024-03-29T11:18:34.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30f8c5c640 1 -- 172.21.15.105:0/1433646329 learned_addr learned my addr 172.21.15.105:0/1433646329 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:34.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 client.?.objecter ms_handle_connect 0x7f30f40b0770 2024-03-29T11:18:34.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:34.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244032270 0 0) 0x7f30f407deb0 con 0x7f30f40b0770 2024-03-29T11:18:34.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.981+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d8004eb0 con 0x7f30f40b0770 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 client.?.objecter ms_handle_connect 0x7f30f40ac5a0 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200220621 0 0) 0x7f30d8004eb0 con 0x7f30f40b0770 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d8005300 con 0x7f30f40b0770 2024-03-29T11:18:34.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f30e4001bc0 con 0x7f30f40b0770 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069446221 0 0) 0x7f30f41e9d30 con 0x7f30f40ac5a0 2024-03-29T11:18:34.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:18:34.962 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.982+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d8005880 con 0x7f30f40ac5a0 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1105181664 0 0) 0x7f30d8005880 con 0x7f30f40ac5a0 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d8005d10 con 0x7f30f40ac5a0 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f30ec001bf0 con 0x7f30f40ac5a0 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.966 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988161825 0 0) 0x7f30d8005300 con 0x7f30f40b0770 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 >> v1:172.21.15.105:6790/0 conn(0x7f30f40ac5a0 legacy=0x7f30f41e9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 >> v1:172.21.15.110:6789/0 conn(0x7f30f414a700 legacy=0x7f30f41f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 monclient: found mon.a 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f41e9ec0 con 0x7f30f40b0770 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:34.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 10 monclient: _renew_subs 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f41476c0 con 0x7f30f40b0770 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 10 monclient: _renew_subs 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f407c280 con 0x7f30f40b0770 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.983+0000 7f30fb6c1640 1 librados: init done 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30fb6c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30fb6c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30fb6c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f40ab560 con 0x7f30f40b0770 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e4004a90 con 0x7f30f40b0770 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.985+0000 7f30e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:34.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.986+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f30e4001bc0 con 0x7f30f40b0770 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.986+0000 7f30e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.986+0000 7f30e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.986+0000 7f30e9ffb640 10 monclient: dump: 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:34.969 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.986+0000 7f30eb7fe640 4 set_mon_vals no callback set 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f30e401aa70 con 0x7f30f40b0770 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30f945d640 10 monclient: get_auth_request con 0x7f30d806eaf0 auth_method 0 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7f30e4054400 con 0x7f30f40b0770 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 10 client.4590.objecter ms_dispatch 0x7f30f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.987+0000 7f30e9ffb640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T11:18:34.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.989+0000 7f30e9ffb640 10 client.4590.objecter ms_handle_connect 0x7f30d806eaf0 2024-03-29T11:18:34.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.995+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30e40506a0 con 0x7f30f40b0770 2024-03-29T11:18:34.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.995+0000 7f30e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:34.995+0000 7f30e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:35.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.323+0000 7f30fb6c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.323+0000 7f30fb6c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.324+0000 7f30fb6c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:35.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.324+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30f40ab560 con 0x7f30f40b0770 2024-03-29T11:18:35.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.325+0000 7f30e9ffb640 1 -- 172.21.15.105:0/1433646329 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7f30e4050840 con 0x7f30f40b0770 2024-03-29T11:18:35.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.325+0000 7f30e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.325+0000 7f30e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:35.304 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:35.304 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:35.305 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.328+0000 7f30fb6c1640 10 librados: watch_flush enter 2024-03-29T11:18:35.305 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.328+0000 7f30fb6c1640 10 librados: watch_flush exit 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.328+0000 7f30fb6c1640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.328+0000 7f30fb6c1640 10 client.4590.objecter successfully canceled tick 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 >> v1:172.21.15.110:6816/1087970004 conn(0x7f30d806eaf0 legacy=0x7f30d8070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 10 monclient: shutdown 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:35.306 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 >> v1:172.21.15.105:6789/0 conn(0x7f30f40b0770 legacy=0x7f30f41ec2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.307 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 shutdown_connections 2024-03-29T11:18:35.307 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 >> 172.21.15.105:0/1433646329 conn(0x7f30f406b660 msgr2=0x7f30f406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:35.307 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 shutdown_connections 2024-03-29T11:18:35.307 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 -- 172.21.15.105:0/1433646329 wait complete. 2024-03-29T11:18:35.307 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:35.329+0000 7f30fb6c1640 1 librados: shutdown 2024-03-29T11:18:35.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:42.131 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:42.276 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.299+0000 7f1b45e72640 10 monclient: get_monmap_and_config 2024-03-29T11:18:42.276 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.299+0000 7f1b45e72640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.299+0000 7f1b45e72640 10 monclient: monmap: 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:42.300385+0000 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:42.300385+0000 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:42.277 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:42.278 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:42.278 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.278 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.301+0000 7f1b45e72640 1 Processor -- start 2024-03-29T11:18:42.279 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.301+0000 7f1b45e72640 1 -- start start 2024-03-29T11:18:42.279 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.301+0000 7f1b45e72640 10 monclient: init 2024-03-29T11:18:42.279 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-a con 0x7f1b400b4920 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400b4200 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4013f9a0 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400b0930 con 0x7f1b400b4920 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400af460 con 0x7f1b4013f9a0 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400aeda0 con 0x7f1b400b4200 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.280 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.302+0000 7f1b45e72640 10 monclient(hunting): authenticate will time out at 1396.578735s 2024-03-29T11:18:42.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.304+0000 7f1b3ffff640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f1b400b4200 0x7f1b400b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:49168/0 (socket says 172.21.15.105:49168) 2024-03-29T11:18:42.282 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3ffff640 1 -- 172.21.15.105:0/3667232931 learned_addr learned my addr 172.21.15.105:0/3667232931 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:42.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245356402 0 0) 0x7f1b400aeda0 con 0x7f1b400b4200 2024-03-29T11:18:42.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.285 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.305+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003520 con 0x7f1b400b4200 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516571886 0 0) 0x7f1b400b0930 con 0x7f1b400b4920 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b400aeda0 con 0x7f1b400b4920 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.306+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1280827379 0 0) 0x7f1b30003520 con 0x7f1b400b4200 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 >> v1:172.21.15.110:6789/0 conn(0x7f1b4013f9a0 legacy=0x7f1b4014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 >> v1:172.21.15.105:6789/0 conn(0x7f1b400b4920 legacy=0x7f1b4013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:18:42.286 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400af980 con 0x7f1b400b4200 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1b2c001690 con 0x7f1b400b4200 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:42.287 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: dump: 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c0018b0 con 0x7f1b400b4200 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.307+0000 7f1b3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b3f7fe640 1 -- 172.21.15.105:0/3667232931 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1b2c001c40 con 0x7f1b400b4200 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.288 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b3f7fe640 10 monclient: dump: 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:42.289 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b45e72640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b45e72640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.308+0000 7f1b45e72640 4 set_mon_vals no callback set 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 10 monclient: shutdown 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 1 -- 172.21.15.105:0/3667232931 >> v1:172.21.15.105:6790/0 conn(0x7f1b400b4200 legacy=0x7f1b400b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 1 -- 172.21.15.105:0/3667232931 shutdown_connections 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 1 -- 172.21.15.105:0/3667232931 >> 172.21.15.105:0/3667232931 conn(0x7f1b4006d230 msgr2=0x7f1b40066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 1 -- 172.21.15.105:0/3667232931 shutdown_connections 2024-03-29T11:18:42.293 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.309+0000 7f1b45e72640 1 -- 172.21.15.105:0/3667232931 wait complete. 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.310+0000 7f1b45e72640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.310+0000 7f1b45e72640 10 monclient: monmap: 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 librados: starting msgr at 2024-03-29T11:18:42.294 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 librados: starting objecter 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 Processor -- start 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 -- start start 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 librados: setting wanted keys 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 1 librados: calling monclient init 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.311+0000 7f1b45e72640 10 monclient: init 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-a con 0x7f1b400b4200 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400b4920 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4013f9a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b40145680 con 0x7f1b400b4200 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b40145350 con 0x7f1b400b4920 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4006e450 con 0x7f1b4013f9a0 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.295 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45e72640 10 monclient(hunting): authenticate will time out at 1396.588472s 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.312+0000 7f1b45671640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f1b4013f9a0 0x7f1b401e5cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:54696/0 (socket says 172.21.15.105:54696) 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b45671640 1 -- 172.21.15.105:0/173586498 learned_addr learned my addr 172.21.15.105:0/173586498 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713865852 0 0) 0x7f1b4006e450 con 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.313+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24004eb0 con 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068832414 0 0) 0x7f1b24004eb0 con 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b4006e450 con 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1b38001ad0 con 0x7f1b4013f9a0 2024-03-29T11:18:42.296 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 10 monclient(hunting): dump: 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:42.297 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 684581119 0 0) 0x7f1b4006e450 con 0x7f1b4013f9a0 2024-03-29T11:18:42.298 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.314+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 >> v1:172.21.15.110:6789/0 conn(0x7f1b400b4920 legacy=0x7f1b401df0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.316+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 >> v1:172.21.15.105:6789/0 conn(0x7f1b400b4200 legacy=0x7f1b401de9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 1 monclient: found mon.c 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4006e730 con 0x7f1b4013f9a0 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:18:42.301 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 10 monclient: _renew_subs 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b40144840 con 0x7f1b4013f9a0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 10 monclient: _renew_subs 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b40142340 con 0x7f1b4013f9a0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.317+0000 7f1b45e72640 1 librados: init done 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.318+0000 7f1b45e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.318+0000 7f1b45e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.318+0000 7f1b45e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4004f730 con 0x7f1b4013f9a0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b38001c30 con 0x7f1b4013f9a0 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.302 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f1b38004d50 con 0x7f1b4013f9a0 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3d7fa640 10 monclient: dump: 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.319+0000 7f1b3effd640 4 set_mon_vals no callback set 2024-03-29T11:18:42.303 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.320+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f1b38004f30 con 0x7f1b4013f9a0 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.321+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7f1b38084d90 con 0x7f1b4013f9a0 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.321+0000 7f1b3d7fa640 10 client.4667.objecter ms_dispatch 0x7f1b40000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.321+0000 7f1b3d7fa640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.321+0000 7f1b3d7fa640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.321+0000 7f1b3d7fa640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.323+0000 7f1b44e70640 10 monclient: get_auth_request con 0x7f1b2406dc40 auth_method 0 2024-03-29T11:18:42.304 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.324+0000 7f1b3d7fa640 10 client.4667.objecter ms_handle_connect 0x7f1b2406dc40 2024-03-29T11:18:42.305 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.328+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b380504a0 con 0x7f1b4013f9a0 2024-03-29T11:18:42.305 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.328+0000 7f1b3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.305 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.328+0000 7f1b3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:42.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.654+0000 7f1b45e72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.655+0000 7f1b45e72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.655+0000 7f1b45e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:42.632 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.655+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b4013e530 con 0x7f1b4013f9a0 2024-03-29T11:18:42.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.656+0000 7f1b3d7fa640 1 -- 172.21.15.105:0/173586498 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7f1b380525a0 con 0x7f1b4013f9a0 2024-03-29T11:18:42.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.656+0000 7f1b3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.633 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.656+0000 7f1b3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:42.634 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:42.635 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 10 librados: watch_flush enter 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 10 librados: watch_flush exit 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 >> v1:172.21.15.110:6816/1087970004 conn(0x7f1b2406dc40 legacy=0x7f1b24070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 10 monclient: shutdown 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.661+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 >> v1:172.21.15.105:6790/0 conn(0x7f1b4013f9a0 legacy=0x7f1b401e5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.663+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 shutdown_connections 2024-03-29T11:18:42.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.663+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 >> 172.21.15.105:0/173586498 conn(0x7f1b4006d230 msgr2=0x7f1b400b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.663+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 shutdown_connections 2024-03-29T11:18:42.640 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.663+0000 7f1b45e72640 1 -- 172.21.15.105:0/173586498 wait complete. 2024-03-29T11:18:42.641 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:42.664+0000 7f1b45e72640 1 librados: shutdown 2024-03-29T11:18:42.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:49.466 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.636+0000 7efdc9874640 10 monclient: get_monmap_and_config 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.636+0000 7efdc9874640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.636+0000 7efdc9874640 10 monclient: monmap: 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:49.637936+0000 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:49.637936+0000 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:49.614 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:49.615 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.638+0000 7efdc9874640 1 Processor -- start 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.638+0000 7efdc9874640 1 -- start start 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.638+0000 7efdc9874640 10 monclient: init 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-a con 0x7efdbc006e30 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-c con 0x7efdbc035e00 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:49.619 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-b con 0x7efdbc0c9670 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc035970 con 0x7efdbc006e30 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc036280 con 0x7efdbc035e00 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc035640 con 0x7efdbc0c9670 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.639+0000 7efdc9874640 10 monclient(hunting): authenticate will time out at 1403.915325s 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc9073640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7efdbc0c9670 0x7efdbc0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:43588/0 (socket says 172.21.15.105:43588) 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc9073640 1 -- 172.21.15.105:0/2808790798 learned_addr learned my addr 172.21.15.105:0/2808790798 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866370964 0 0) 0x7efdbc035970 con 0x7efdbc006e30 2024-03-29T11:18:49.620 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb4003520 con 0x7efdbc006e30 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.642+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665201337 0 0) 0x7efdbc035640 con 0x7efdbc0c9670 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:18:49.621 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdbc035970 con 0x7efdbc0c9670 2024-03-29T11:18:49.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960516467 0 0) 0x7efdb4003520 con 0x7efdbc006e30 2024-03-29T11:18:49.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 >> v1:172.21.15.105:6790/0 conn(0x7efdbc0c9670 legacy=0x7efdbc0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.622 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 >> v1:172.21.15.110:6789/0 conn(0x7efdbc035e00 legacy=0x7efdbc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 monclient: found mon.noname-a 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.105:6789/0 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc034e20 con 0x7efdbc006e30 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7efdb0001690 con 0x7efdbc006e30 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: dump: 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:49.623 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.0 v1:172.21.15.105:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb00018b0 con 0x7efdbc006e30 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 1 -- 172.21.15.105:0/2808790798 <== mon.0 v1:172.21.15.105:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7efdb0001c40 con 0x7efdbc006e30 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.643+0000 7efdc37fe640 10 monclient: dump: 2024-03-29T11:18:49.624 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.644+0000 7efdc9874640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.644+0000 7efdc9874640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.644+0000 7efdc9874640 4 set_mon_vals no callback set 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.645+0000 7efdc9874640 10 monclient: shutdown 2024-03-29T11:18:49.625 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.645+0000 7efdc9874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:49.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.645+0000 7efdc9874640 1 -- 172.21.15.105:0/2808790798 >> v1:172.21.15.105:6789/0 conn(0x7efdbc006e30 legacy=0x7efdbc0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.646+0000 7efdc9874640 1 -- 172.21.15.105:0/2808790798 shutdown_connections 2024-03-29T11:18:49.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.646+0000 7efdc9874640 1 -- 172.21.15.105:0/2808790798 >> 172.21.15.105:0/2808790798 conn(0x7efdbc005c00 msgr2=0x7efdbc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:49.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.646+0000 7efdc9874640 1 -- 172.21.15.105:0/2808790798 shutdown_connections 2024-03-29T11:18:49.626 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.646+0000 7efdc9874640 1 -- 172.21.15.105:0/2808790798 wait complete. 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.647+0000 7efdc9874640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.647+0000 7efdc9874640 10 monclient: monmap: 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.647+0000 7efdc9874640 1 librados: starting msgr at 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.647+0000 7efdc9874640 1 librados: starting objecter 2024-03-29T11:18:49.627 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.648+0000 7efdc9874640 1 Processor -- start 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.648+0000 7efdc9874640 1 -- start start 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.648+0000 7efdc9874640 1 librados: setting wanted keys 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.648+0000 7efdc9874640 1 librados: calling monclient init 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.648+0000 7efdc9874640 10 monclient: init 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.628 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-a con 0x7efdbc006e30 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-b con 0x7efdbc035e00 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient(hunting): picked mon.noname-c con 0x7efdbc0c9670 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc02e3a0 con 0x7efdbc006e30 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc02f730 con 0x7efdbc035e00 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc02e5d0 con 0x7efdbc0c9670 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.649+0000 7efdc9874640 10 monclient(hunting): authenticate will time out at 1403.925126s 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.651+0000 7efdc9073640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7efdbc0c9670 0x7efdbc161620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:48396/0 (socket says 172.21.15.105:48396) 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.651+0000 7efdc9073640 1 -- 172.21.15.105:0/2995538786 learned_addr learned my addr 172.21.15.105:0/2995538786 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.651+0000 7efdc17fa640 10 client.?.objecter ms_handle_connect 0x7efdbc0c9670 2024-03-29T11:18:49.629 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.651+0000 7efdc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.652+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925166336 0 0) 0x7efdbc02e5d0 con 0x7efdbc0c9670 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda8004eb0 con 0x7efdbc0c9670 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1935226236 0 0) 0x7efda8004eb0 con 0x7efdbc0c9670 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdbc02e5d0 con 0x7efdbc0c9670 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7efdb8001890 con 0x7efdbc0c9670 2024-03-29T11:18:49.630 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): dump: 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.631 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124017001 0 0) 0x7efdbc02e5d0 con 0x7efdbc0c9670 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.653+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 >> v1:172.21.15.110:6789/0 conn(0x7efdbc035e00 legacy=0x7efdbc0c1ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 >> v1:172.21.15.105:6789/0 conn(0x7efdbc006e30 legacy=0x7efdbc02dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 1 monclient: found mon.c 2024-03-29T11:18:49.634 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.654+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc02fea0 con 0x7efdbc0c9670 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb80019f0 con 0x7efdbc0c9670 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7efdb8004a80 con 0x7efdbc0c9670 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.655+0000 7efdc17fa640 10 monclient: dump: 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.635 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 10 monclient: _renew_subs 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc0300f0 con 0x7efdbc0c9670 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 10 monclient: _renew_subs 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc0c23b0 con 0x7efdbc0c9670 2024-03-29T11:18:49.636 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.656+0000 7efdc9874640 1 librados: init done 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.658+0000 7efdc2ffd640 4 set_mon_vals no callback set 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.658+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7efdb801aa50 con 0x7efdbc0c9670 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc9874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7efdb8084a70 con 0x7efdbc0c9670 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc17fa640 10 client.4673.objecter ms_dispatch 0x7efdc4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc17fa640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:18:49.638 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc17fa640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc9874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc9874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda4005080 con 0x7efdbc0c9670 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc8872640 10 monclient: get_auth_request con 0x7efda806dc90 auth_method 0 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.659+0000 7efdc17fa640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:18:49.639 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.660+0000 7efdc17fa640 10 client.4673.objecter ms_handle_connect 0x7efda806dc90 2024-03-29T11:18:49.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.669+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdb8050180 con 0x7efdbc0c9670 2024-03-29T11:18:49.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.669+0000 7efdc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.647 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.669+0000 7efdc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:49.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.996+0000 7efdc9874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.996+0000 7efdc9874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.996+0000 7efdc9874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:18:49.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:49.996+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda4005330 con 0x7efdbc0c9670 2024-03-29T11:18:49.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.001+0000 7efdc17fa640 1 -- 172.21.15.105:0/2995538786 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7efdb8052280 con 0x7efdbc0c9670 2024-03-29T11:18:49.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.001+0000 7efdc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.978 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.001+0000 7efdc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:49.978 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:49.978 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:49.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 10 librados: watch_flush enter 2024-03-29T11:18:49.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 10 librados: watch_flush exit 2024-03-29T11:18:49.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:18:49.981 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 >> v1:172.21.15.110:6816/1087970004 conn(0x7efda806dc90 legacy=0x7efda8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 10 monclient: shutdown 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 >> v1:172.21.15.105:6790/0 conn(0x7efdbc0c9670 legacy=0x7efdbc161620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 shutdown_connections 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 >> 172.21.15.105:0/2995538786 conn(0x7efdbc005c00 msgr2=0x7efdbc0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 shutdown_connections 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 -- 172.21.15.105:0/2995538786 wait complete. 2024-03-29T11:18:49.982 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:50.004+0000 7efdc9874640 1 librados: shutdown 2024-03-29T11:18:50.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:56.806 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.969+0000 7fbf07a86640 10 monclient: get_monmap_and_config 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.969+0000 7fbf07a86640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.970+0000 7fbf07a86640 10 monclient: monmap: 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:18:56.971052+0000 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:18:56.971052+0000 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.947 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:56.948 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:18:56.948 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:18:56.948 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.949 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.971+0000 7fbf07a86640 1 Processor -- start 2024-03-29T11:18:56.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.972+0000 7fbf07a86640 1 -- start start 2024-03-29T11:18:56.953 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.972+0000 7fbf07a86640 10 monclient: init 2024-03-29T11:18:56.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.954 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-c con 0x7fbf000b0770 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0014a700 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-a con 0x7fbf000ac5a0 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000aba70 con 0x7fbf000ac5a0 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000aecb0 con 0x7fbf000b0770 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000af150 con 0x7fbf0014a700 2024-03-29T11:18:56.955 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf07a86640 10 monclient(hunting): authenticate will time out at 1411.249201s 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf06023640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fbf000ac5a0 0x7fbf00151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:56148/0 (socket says 172.21.15.105:56148) 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf06023640 1 -- 172.21.15.105:0/3249958538 learned_addr learned my addr 172.21.15.105:0/3249958538 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.973+0000 7fbf05822640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbf000b0770 0x7fbf00149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:37562/0 (socket says 172.21.15.105:37562) 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737387435 0 0) 0x7fbf000af150 con 0x7fbf0014a700 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.956 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec003520 con 0x7fbf0014a700 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721599850 0 0) 0x7fbf000aba70 con 0x7fbf000ac5a0 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.957 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf000af150 con 0x7fbf000ac5a0 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040133743 0 0) 0x7fbeec003520 con 0x7fbf0014a700 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 >> v1:172.21.15.110:6789/0 conn(0x7fbf000b0770 legacy=0x7fbf00149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 >> v1:172.21.15.105:6789/0 conn(0x7fbf000ac5a0 legacy=0x7fbf00151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 monclient: found mon.noname-b 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000ac9d0 con 0x7fbf0014a700 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.958 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf07a86640 5 monclient: authenticate success, global_id 4676 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbef00015c0 con 0x7fbf0014a700 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf07a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf04820640 10 monclient: dump: 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:56.959 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.974+0000 7fbf07a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef0001720 con 0x7fbf0014a700 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf07a86640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf07a86640 4 set_mon_vals no callback set 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 1 -- 172.21.15.105:0/3249958538 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbef0001b30 con 0x7fbf0014a700 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf04820640 10 monclient: dump: 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.960 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf07a86640 10 monclient: shutdown 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf07a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.961 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.976+0000 7fbf07a86640 1 -- 172.21.15.105:0/3249958538 >> v1:172.21.15.105:6790/0 conn(0x7fbf0014a700 legacy=0x7fbf000abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.977+0000 7fbf07a86640 1 -- 172.21.15.105:0/3249958538 shutdown_connections 2024-03-29T11:18:56.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.977+0000 7fbf07a86640 1 -- 172.21.15.105:0/3249958538 >> 172.21.15.105:0/3249958538 conn(0x7fbf0006b660 msgr2=0x7fbf0006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.967 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.977+0000 7fbf07a86640 1 -- 172.21.15.105:0/3249958538 shutdown_connections 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.977+0000 7fbf07a86640 1 -- 172.21.15.105:0/3249958538 wait complete. 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 10 monclient: monmap: 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 librados: starting msgr at 2024-03-29T11:18:56.968 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 librados: starting objecter 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 Processor -- start 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 -- start start 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 librados: setting wanted keys 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 1 librados: calling monclient init 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.979+0000 7fbf07a86640 10 monclient: init 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-c con 0x7fbf000ac5a0 addr v1:172.21.15.105:6790/0 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-b con 0x7fbf000b0770 addr v1:172.21.15.110:6789/0 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient(hunting): picked mon.noname-a con 0x7fbf0014a700 addr v1:172.21.15.105:6789/0 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf001239d0 con 0x7fbf0014a700 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf00128210 con 0x7fbf000b0770 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf00127440 con 0x7fbf000ac5a0 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.969 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.980+0000 7fbf07a86640 10 monclient(hunting): authenticate will time out at 1411.256794s 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbf06023640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fbf0014a700 0x7fbf001db8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:56634/0 (socket says 172.21.15.105:56634) 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbf06023640 1 -- 172.21.15.105:0/389009281 learned_addr learned my addr 172.21.15.105:0/389009281 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 client.?.objecter ms_handle_connect 0x7fbf000ac5a0 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 client.?.objecter ms_handle_connect 0x7fbf0014a700 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751578652 0 0) 0x7fbf000aecb0 con 0x7fbf000ac5a0 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee4004eb0 con 0x7fbf000ac5a0 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766868021 0 0) 0x7fbf001239d0 con 0x7fbf0014a700 2024-03-29T11:18:56.970 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.981+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf000aecb0 con 0x7fbf0014a700 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193675155 0 0) 0x7fbf000aecb0 con 0x7fbf0014a700 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf001239d0 con 0x7fbf0014a700 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbef8001cb0 con 0x7fbf0014a700 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient(hunting): dump: 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.971 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 client.?.objecter ms_handle_connect 0x7fbf000b0770 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1936092036 0 0) 0x7fbf001239d0 con 0x7fbf0014a700 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 >> v1:172.21.15.105:6790/0 conn(0x7fbf000ac5a0 legacy=0x7fbf00126050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 >> v1:172.21.15.110:6789/0 conn(0x7fbf000b0770 legacy=0x7fbf00144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 monclient: found mon.a 2024-03-29T11:18:56.972 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf00128440 con 0x7fbf0014a700 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 10 monclient: _renew_subs 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf00145680 con 0x7fbf0014a700 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 10 monclient: _renew_subs 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf00126df0 con 0x7fbf0014a700 2024-03-29T11:18:56.973 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbf07a86640 1 librados: init done 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.982+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef8004b60 con 0x7fbf0014a700 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef7fff640 4 set_mon_vals no callback set 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fbef8004e70 con 0x7fbf0014a700 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.983+0000 7fbef67fc640 10 monclient: dump: 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:18:56.974 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.984+0000 7fbf07a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.984+0000 7fbf07a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.984+0000 7fbf07a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.985+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee8005080 con 0x7fbf0014a700 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.986+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fbef801ae40 con 0x7fbf0014a700 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.988+0000 7fbf05822640 10 monclient: get_auth_request con 0x7fbee406dc90 auth_method 0 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.988+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7fbef8084d70 con 0x7fbf0014a700 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.989+0000 7fbef67fc640 10 client.4602.objecter ms_dispatch 0x7fbf00000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.989+0000 7fbef67fc640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.989+0000 7fbef67fc640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.989+0000 7fbef67fc640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:18:56.975 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.989+0000 7fbef67fc640 10 client.4602.objecter ms_handle_connect 0x7fbee406dc90 2024-03-29T11:18:56.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.995+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbef80505c0 con 0x7fbf0014a700 2024-03-29T11:18:56.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.995+0000 7fbef67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.976 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:56.995+0000 7fbef67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.362+0000 7fbf07a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.362+0000 7fbf07a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.362+0000 7fbf07a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:18:57.339 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.362+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbee8005330 con 0x7fbf0014a700 2024-03-29T11:18:57.340 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.364+0000 7fbef67fc640 1 -- 172.21.15.105:0/389009281 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7fbef80526c0 con 0x7fbf0014a700 2024-03-29T11:18:57.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.364+0000 7fbef67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.341 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.364+0000 7fbef67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.341 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:18:57.341 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 10 librados: watch_flush enter 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 10 librados: watch_flush exit 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 >> v1:172.21.15.110:6816/1087970004 conn(0x7fbee406dc90 legacy=0x7fbee4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 10 monclient: shutdown 2024-03-29T11:18:57.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 >> v1:172.21.15.105:6789/0 conn(0x7fbf0014a700 legacy=0x7fbf001db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 shutdown_connections 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 >> 172.21.15.105:0/389009281 conn(0x7fbf0006b660 msgr2=0x7fbf000abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 shutdown_connections 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.367+0000 7fbf07a86640 1 -- 172.21.15.105:0/389009281 wait complete. 2024-03-29T11:18:57.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:18:57.368+0000 7fbf07a86640 1 librados: shutdown 2024-03-29T11:18:57.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:04.166 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:04.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.356+0000 7f2dcb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:04.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.356+0000 7f2dcb59e640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.334 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.356+0000 7f2dcb59e640 10 monclient: monmap: 2024-03-29T11:19:04.334 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:19:04.357344+0000 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:19:04.357344+0000 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.335 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.357+0000 7f2dcb59e640 1 Processor -- start 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.357+0000 7f2dcb59e640 1 -- start start 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.357+0000 7f2dcb59e640 10 monclient: init 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dcc0a9750 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dcc0ae220 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dcc13b9d0 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:04.342 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc142490 con 0x7f2dcc0a9750 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0ac130 con 0x7f2dcc0ae220 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0ac410 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.358+0000 7f2dcb59e640 10 monclient(hunting): authenticate will time out at 1418.634925s 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.364+0000 7f2dcad9d640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f2dcc13b9d0 0x7f2dcc149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46049/0 (socket says 172.21.15.105:46049) 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.364+0000 7f2dcad9d640 1 -- 172.21.15.105:0/4076850825 learned_addr learned my addr 172.21.15.105:0/4076850825 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460252658 0 0) 0x7f2dcc0ac410 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:19:04.343 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db4003520 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031311075 0 0) 0x7f2db4003520 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.344 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 >> v1:172.21.15.110:6789/0 conn(0x7f2dcc0ae220 legacy=0x7f2dcc13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.365+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 >> v1:172.21.15.105:6789/0 conn(0x7f2dcc0a9750 legacy=0x7f2dcc141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 1 monclient: found mon.noname-b 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc0afa40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2dc4001a10 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: dump: 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:04.345 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dcb59e640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dcb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc4001c30 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 1 -- 172.21.15.105:0/4076850825 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2dc4001f40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.346 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dc959a640 10 monclient: dump: 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dcb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:04.347 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.366+0000 7f2dcb59e640 4 set_mon_vals no callback set 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.369+0000 7f2dcb59e640 10 monclient: shutdown 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.369+0000 7f2dcb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.369+0000 7f2dcb59e640 1 -- 172.21.15.105:0/4076850825 >> v1:172.21.15.105:6790/0 conn(0x7f2dcc13b9d0 legacy=0x7f2dcc149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.369+0000 7f2dcb59e640 1 -- 172.21.15.105:0/4076850825 shutdown_connections 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.369+0000 7f2dcb59e640 1 -- 172.21.15.105:0/4076850825 >> 172.21.15.105:0/4076850825 conn(0x7f2dcc0673e0 msgr2=0x7f2dcc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.348 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.371+0000 7f2dcb59e640 1 -- 172.21.15.105:0/4076850825 shutdown_connections 2024-03-29T11:19:04.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.371+0000 7f2dcb59e640 1 -- 172.21.15.105:0/4076850825 wait complete. 2024-03-29T11:19:04.350 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.351 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 10 monclient: monmap: 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 librados: starting msgr at 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 librados: starting objecter 2024-03-29T11:19:04.352 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 Processor -- start 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 -- start start 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 librados: setting wanted keys 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 1 librados: calling monclient init 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.373+0000 7f2dcb59e640 10 monclient: init 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dcc0a9750 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dcc0ae220 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dcc13b9d0 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc1e1b60 con 0x7f2dcc0ae220 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc13df40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc13e7d0 con 0x7f2dcc0a9750 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.353 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.374+0000 7f2dcb59e640 10 monclient(hunting): authenticate will time out at 1418.650700s 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.377+0000 7f2dca59c640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f2dcc0a9750 0x7f2dcc1e4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:46149/0 (socket says 172.21.15.105:46149) 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dcad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2dcc13b9d0 0x7f2dcc1e1450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:48949/0 (socket says 172.21.15.105:48949) 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dcad9d640 1 -- 172.21.15.105:0/2405975065 learned_addr learned my addr 172.21.15.105:0/2405975065 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dbaffd640 10 client.?.objecter ms_handle_connect 0x7f2dcc0a9750 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dbaffd640 10 client.?.objecter ms_handle_connect 0x7f2dcc13b9d0 2024-03-29T11:19:04.355 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.378+0000 7f2dbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959011242 0 0) 0x7f2dcc13e7d0 con 0x7f2dcc0a9750 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004eb0 con 0x7f2dcc0a9750 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474390944 0 0) 0x7f2dcc13df40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.357 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.359 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:19:04.360 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.379+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac006230 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984297528 0 0) 0x7f2dac006230 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dcc13df40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2dc40018d0 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:04.361 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.383+0000 7f2dbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.384+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1400560246 0 0) 0x7f2dcc13df40 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.384+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 >> v1:172.21.15.105:6790/0 conn(0x7f2dcc0a9750 legacy=0x7f2dcc1e4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.362 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.384+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 >> v1:172.21.15.105:6789/0 conn(0x7f2dcc0ae220 legacy=0x7f2dcc1e0d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 1 monclient: found mon.b 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc13d310 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 5 monclient: authenticate success, global_id 4579 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 10 monclient: _renew_subs 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc13db80 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:04.363 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 10 monclient: _renew_subs 2024-03-29T11:19:04.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:19:04.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc13e460 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.364 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.386+0000 7f2dcb59e640 1 librados: init done 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc4004e10 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2dc4005120 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbaffd640 10 monclient: dump: 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:04.365 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:04.366 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:04.366 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:04.366 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:04.366 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.387+0000 7f2dbbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.389+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f2dc401b0f0 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.390+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7f2dc4084ed0 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.390+0000 7f2dbaffd640 10 client.4579.objecter ms_dispatch 0x7f2dcc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.390+0000 7f2dbaffd640 3 client.4579.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.390+0000 7f2dbaffd640 3 client.4579.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:04.367 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.390+0000 7f2dbaffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-29T11:19:04.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dca59c640 10 monclient: get_auth_request con 0x7f2dac06df60 auth_method 0 2024-03-29T11:19:04.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dcb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dcb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dbaffd640 10 client.4579.objecter ms_handle_connect 0x7f2dac06df60 2024-03-29T11:19:04.368 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dcb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:19:04.369 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.391+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0005080 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.398+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dc40507b0 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.398+0000 7f2dbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.375 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.398+0000 7f2dbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:04.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.761+0000 7f2dcb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.761+0000 7f2dcb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.761+0000 7f2dcb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.110:6789/0 2024-03-29T11:19:04.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.761+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db0005330 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.762+0000 7f2dbaffd640 1 -- 172.21.15.105:0/2405975065 <== mon.1 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7f2dc40526e0 con 0x7f2dcc13b9d0 2024-03-29T11:19:04.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.762+0000 7f2dbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.739 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.762+0000 7f2dbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:04.739 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:19:04.740 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:04.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.765+0000 7f2dcb59e640 10 librados: watch_flush enter 2024-03-29T11:19:04.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.765+0000 7f2dcb59e640 10 librados: watch_flush exit 2024-03-29T11:19:04.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.765+0000 7f2dcb59e640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-29T11:19:04.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.765+0000 7f2dcb59e640 10 client.4579.objecter successfully canceled tick 2024-03-29T11:19:04.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.765+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 >> v1:172.21.15.110:6816/1087970004 conn(0x7f2dac06df60 legacy=0x7f2dac070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 10 monclient: shutdown 2024-03-29T11:19:04.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 >> v1:172.21.15.110:6789/0 conn(0x7f2dcc13b9d0 legacy=0x7f2dcc1e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 shutdown_connections 2024-03-29T11:19:04.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 >> 172.21.15.105:0/2405975065 conn(0x7f2dcc0673e0 msgr2=0x7f2dcc0b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 shutdown_connections 2024-03-29T11:19:04.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 -- 172.21.15.105:0/2405975065 wait complete. 2024-03-29T11:19:04.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:04.766+0000 7f2dcb59e640 1 librados: shutdown 2024-03-29T11:19:04.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:11.567 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:11.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.755+0000 7f6cac73d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:11.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.755+0000 7f6cac73d640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.733 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.755+0000 7f6cac73d640 10 monclient: monmap: 2024-03-29T11:19:11.733 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:19:11.756803+0000 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:19:11.756803+0000 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.756+0000 7f6cac73d640 1 Processor -- start 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.756+0000 7f6cac73d640 1 -- start start 2024-03-29T11:19:11.734 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.756+0000 7f6cac73d640 10 monclient: init 2024-03-29T11:19:11.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.758+0000 7f6cac73d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.735 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.758+0000 7f6cac73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca40a9ac0 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca413ee90 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca40af900 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40afd10 con 0x7f6ca40af900 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40af6f0 con 0x7f6ca413ee90 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40aac80 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.738 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.761+0000 7f6cac73d640 10 monclient(hunting): authenticate will time out at 1426.037274s 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.762+0000 7f6caa4d9640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7f6ca40a9ac0 0x7f6ca40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:55217/0 (socket says 172.21.15.105:55217) 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.762+0000 7f6caa4d9640 1 -- 172.21.15.105:0/2401327730 learned_addr learned my addr 172.21.15.105:0/2401327730 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085976559 0 0) 0x7f6ca40aac80 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.741 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90003520 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093775579 0 0) 0x7f6ca40afd10 con 0x7f6ca40af900 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:19:11.742 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.763+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca40aac80 con 0x7f6ca40af900 2024-03-29T11:19:11.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.764+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712488688 0 0) 0x7f6c90003520 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.764+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 >> v1:172.21.15.110:6789/0 conn(0x7f6ca413ee90 legacy=0x7f6ca413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:11.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.764+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 >> v1:172.21.15.105:6789/0 conn(0x7f6ca40af900 legacy=0x7f6ca4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.743 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 1 monclient: found mon.noname-b 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40aaf60 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6ca0004c90 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: dump: 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.744 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6cac73d640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6cac73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca0004eb0 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 1 -- 172.21.15.105:0/2401327730 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6ca0001340 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.765+0000 7f6ca94d7640 10 monclient: dump: 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.745 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.766+0000 7f6cac73d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:11.746 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.766+0000 7f6cac73d640 4 set_mon_vals no callback set 2024-03-29T11:19:11.747 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.767+0000 7f6cac73d640 10 monclient: shutdown 2024-03-29T11:19:11.747 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.767+0000 7f6cac73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.767+0000 7f6cac73d640 1 -- 172.21.15.105:0/2401327730 >> v1:172.21.15.105:6790/0 conn(0x7f6ca40a9ac0 legacy=0x7f6ca40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.768+0000 7f6cac73d640 1 -- 172.21.15.105:0/2401327730 shutdown_connections 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.768+0000 7f6cac73d640 1 -- 172.21.15.105:0/2401327730 >> 172.21.15.105:0/2401327730 conn(0x7f6ca40b35e0 msgr2=0x7f6ca40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.768+0000 7f6cac73d640 1 -- 172.21.15.105:0/2401327730 shutdown_connections 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.768+0000 7f6cac73d640 1 -- 172.21.15.105:0/2401327730 wait complete. 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.769+0000 7f6cac73d640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.769+0000 7f6cac73d640 10 monclient: monmap: 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:11.748 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.770+0000 7f6cac73d640 1 librados: starting msgr at 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.770+0000 7f6cac73d640 1 librados: starting objecter 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.770+0000 7f6cac73d640 1 Processor -- start 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.770+0000 7f6cac73d640 1 -- start start 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.771+0000 7f6cac73d640 1 librados: setting wanted keys 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.771+0000 7f6cac73d640 1 librados: calling monclient init 2024-03-29T11:19:11.749 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.771+0000 7f6cac73d640 10 monclient: init 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca40a9ac0 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca40af900 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca413ee90 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:11.750 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca406f140 con 0x7f6ca413ee90 2024-03-29T11:19:11.751 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40705b0 con 0x7f6ca40af900 2024-03-29T11:19:11.751 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca4142240 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.751 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.751 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.772+0000 7f6cac73d640 10 monclient(hunting): authenticate will time out at 1426.048972s 2024-03-29T11:19:11.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.776+0000 7f6caacda640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f6ca413ee90 0x7f6ca41d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:38068/0 (socket says 172.21.15.105:38068) 2024-03-29T11:19:11.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.776+0000 7f6caacda640 1 -- 172.21.15.105:0/3888790561 learned_addr learned my addr 172.21.15.105:0/3888790561 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:11.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.776+0000 7f6c9affd640 10 client.?.objecter ms_handle_connect 0x7f6ca413ee90 2024-03-29T11:19:11.753 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.776+0000 7f6c9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 client.?.objecter ms_handle_connect 0x7f6ca40a9ac0 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993859093 0 0) 0x7f6ca406f140 con 0x7f6ca413ee90 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c88004eb0 con 0x7f6ca413ee90 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70338420 0 0) 0x7f6ca4142240 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.756 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:19:11.757 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca406f140 con 0x7f6ca40a9ac0 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2937270648 0 0) 0x7f6c88004eb0 con 0x7f6ca413ee90 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ca4142240 con 0x7f6ca413ee90 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6c9c001d50 con 0x7f6ca413ee90 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): dump: 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.758 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165457990 0 0) 0x7f6ca4142240 con 0x7f6ca413ee90 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 >> v1:172.21.15.105:6790/0 conn(0x7f6ca40a9ac0 legacy=0x7f6ca406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.759 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.779+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 >> v1:172.21.15.110:6789/0 conn(0x7f6ca40af900 legacy=0x7f6ca4139760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 1 monclient: found mon.a 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca41423d0 con 0x7f6ca413ee90 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.761 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6cac73d640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6cac73d640 10 monclient: _renew_subs 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6cac73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c9c003970 con 0x7f6ca413ee90 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6cac73d640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca4139e70 con 0x7f6ca413ee90 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.782+0000 7f6c9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f6c9c003d80 con 0x7f6ca413ee90 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6c9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6c9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6c9affd640 10 monclient: dump: 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:11.762 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6cac73d640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6ca8cd6640 4 set_mon_vals no callback set 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6cac73d640 10 monclient: _renew_subs 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6cac73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6cac73d640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca41d3070 con 0x7f6ca413ee90 2024-03-29T11:19:11.763 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.783+0000 7f6cac73d640 1 librados: init done 2024-03-29T11:19:11.766 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.784+0000 7f6c98ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.785+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f6c9c003ee0 con 0x7f6ca413ee90 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.786+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7f6c9c084e70 con 0x7f6ca413ee90 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.786+0000 7f6c9affd640 10 client.4611.objecter ms_dispatch 0x7f6ca4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.786+0000 7f6c9affd640 3 client.4611.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.786+0000 7f6c9affd640 3 client.4611.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.786+0000 7f6c9affd640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.787+0000 7f6c98ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.787+0000 7f6c98ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.787+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca413bd00 con 0x7f6ca413ee90 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.787+0000 7f6caa4d9640 10 monclient: get_auth_request con 0x7f6c88072060 auth_method 0 2024-03-29T11:19:11.767 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.788+0000 7f6c9affd640 10 client.4611.objecter ms_handle_connect 0x7f6c88072060 2024-03-29T11:19:11.774 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.797+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ca413bd00 con 0x7f6ca413ee90 2024-03-29T11:19:11.774 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.797+0000 7f6c9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.774 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:11.797+0000 7f6c9affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:12.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.154+0000 7f6c98ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.154+0000 7f6c98ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.131 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.154+0000 7f6c98ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.105:6789/0 2024-03-29T11:19:12.132 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.154+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 --> v1:172.21.15.105:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ca413bd00 con 0x7f6ca413ee90 2024-03-29T11:19:12.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.157+0000 7f6c9affd640 1 -- 172.21.15.105:0/3888790561 <== mon.0 v1:172.21.15.105:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7f6ca413bd00 con 0x7f6ca413ee90 2024-03-29T11:19:12.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.157+0000 7f6c9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:12.134 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.157+0000 7f6c9affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:12.135 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:19:12.136 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:12.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.161+0000 7f6c98ff9640 10 librados: watch_flush enter 2024-03-29T11:19:12.138 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 10 librados: watch_flush exit 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 10 client.4611.objecter successfully canceled tick 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 >> v1:172.21.15.110:6816/1087970004 conn(0x7f6c88072060 legacy=0x7f6c88074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 10 monclient: shutdown 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 >> v1:172.21.15.105:6789/0 conn(0x7f6ca413ee90 legacy=0x7f6ca41d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 shutdown_connections 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 >> 172.21.15.105:0/3888790561 conn(0x7f6ca40b35e0 msgr2=0x7f6ca413b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:12.139 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 shutdown_connections 2024-03-29T11:19:12.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 -- 172.21.15.105:0/3888790561 wait complete. 2024-03-29T11:19:12.140 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:12.162+0000 7f6c98ff9640 1 librados: shutdown 2024-03-29T11:19:12.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:18.965 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:19.100 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.123+0000 7fb31fae4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:19.100 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.123+0000 7fb31fae4640 10 monclient: build_initial_monmap 2024-03-29T11:19:19.100 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.123+0000 7fb31fae4640 10 monclient: monmap: 2024-03-29T11:19:19.100 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:19:19.124478+0000 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:19:19.124478+0000 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:19:19.101 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.124+0000 7fb31fae4640 1 Processor -- start 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.124+0000 7fb31fae4640 1 -- start start 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.124+0000 7fb31fae4640 10 monclient: init 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-c con 0x7fb3180b0500 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-a con 0x7fb3180b08e0 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-b con 0x7fb3180aeca0 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb318140620 con 0x7fb3180b08e0 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31813daf0 con 0x7fb3180b0500 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180ae180 con 0x7fb3180aeca0 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.125+0000 7fb31fae4640 10 monclient(hunting): authenticate will time out at 1433.401923s 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31e081640 1 --1- >> v1:172.21.15.105:6790/0 conn(0x7fb3180aeca0 0x7fb31814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6790/0 says I am v1:172.21.15.105:57946/0 (socket says 172.21.15.105:57946) 2024-03-29T11:19:19.103 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31e081640 1 -- 172.21.15.105:0/240642940 learned_addr learned my addr 172.21.15.105:0/240642940 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509400431 0 0) 0x7fb3180ae180 con 0x7fb3180aeca0 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb304003520 con 0x7fb3180aeca0 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339427352 0 0) 0x7fb318140620 con 0x7fb3180b08e0 2024-03-29T11:19:19.105 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.126+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3180ae180 con 0x7fb3180b08e0 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.127+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643279586 0 0) 0x7fb304003520 con 0x7fb3180aeca0 2024-03-29T11:19:19.106 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.127+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 >> v1:172.21.15.110:6789/0 conn(0x7fb3180b0500 legacy=0x7fb31813ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 >> v1:172.21.15.105:6789/0 conn(0x7fb3180b08e0 legacy=0x7fb31813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: _finish_hunting 0 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 monclient: found mon.noname-b 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180accf0 con 0x7fb3180aeca0 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: _finish_auth 0 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: _check_auth_tickets 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb310001a10 con 0x7fb3180aeca0 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:19.109 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: dump: 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31fae4640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31fae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb310001b70 con 0x7fb3180aeca0 2024-03-29T11:19:19.110 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31fae4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31fae4640 4 set_mon_vals no callback set 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 1 -- 172.21.15.105:0/240642940 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb310001fc0 con 0x7fb3180aeca0 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.129+0000 7fb31c87e640 10 monclient: dump: 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:19.111 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.130+0000 7fb31fae4640 10 monclient: shutdown 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.130+0000 7fb31fae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.130+0000 7fb31fae4640 1 -- 172.21.15.105:0/240642940 >> v1:172.21.15.105:6790/0 conn(0x7fb3180aeca0 legacy=0x7fb31814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.130+0000 7fb31fae4640 1 -- 172.21.15.105:0/240642940 shutdown_connections 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.130+0000 7fb31fae4640 1 -- 172.21.15.105:0/240642940 >> 172.21.15.105:0/240642940 conn(0x7fb31806b660 msgr2=0x7fb31806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.131+0000 7fb31fae4640 1 -- 172.21.15.105:0/240642940 shutdown_connections 2024-03-29T11:19:19.112 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.131+0000 7fb31fae4640 1 -- 172.21.15.105:0/240642940 wait complete. 2024-03-29T11:19:19.117 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.132+0000 7fb31fae4640 10 monclient: build_initial_monmap 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.132+0000 7fb31fae4640 10 monclient: monmap: 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 librados: starting msgr at 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 librados: starting objecter 2024-03-29T11:19:19.118 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 Processor -- start 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 -- start start 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 librados: setting wanted keys 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 1 librados: calling monclient init 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.133+0000 7fb31fae4640 10 monclient: init 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-a con 0x7fb3180aeca0 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-b con 0x7fb3180b0500 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient(hunting): picked mon.noname-c con 0x7fb3181df200 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3181dbdc0 con 0x7fb3180aeca0 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3181dc2e0 con 0x7fb3180b0500 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3181dc470 con 0x7fb3181df200 2024-03-29T11:19:19.119 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31fae4640 10 monclient(hunting): authenticate will time out at 1433.410851s 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.134+0000 7fb31d880640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7fb3180aeca0 0x7fb3181db5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:44772/0 (socket says 172.21.15.105:44772) 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb31d880640 1 -- 172.21.15.105:0/1747568845 learned_addr learned my addr 172.21.15.105:0/1747568845 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 client.?.objecter ms_handle_connect 0x7fb3180aeca0 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 client.?.objecter ms_handle_connect 0x7fb3181df200 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676543853 0 0) 0x7fb3181dbdc0 con 0x7fb3180aeca0 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:19:19.120 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc004eb0 con 0x7fb3180aeca0 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874496838 0 0) 0x7fb3181dc470 con 0x7fb3181df200 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.135+0000 7fb30e7fc640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3181dbdc0 con 0x7fb3181df200 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083810899 0 0) 0x7fb3181dbdc0 con 0x7fb3181df200 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3181dc470 con 0x7fb3181df200 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb310001e80 con 0x7fb3181df200 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.121 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074749690 0 0) 0x7fb2fc004eb0 con 0x7fb3180aeca0 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3181dbdc0 con 0x7fb3180aeca0 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb314001870 con 0x7fb3180aeca0 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:19.122 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.136+0000 7fb30e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.137+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2698367097 0 0) 0x7fb3181dc470 con 0x7fb3181df200 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.137+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 >> v1:172.21.15.110:6789/0 conn(0x7fb3180b0500 legacy=0x7fb3181deaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.137+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 >> v1:172.21.15.105:6789/0 conn(0x7fb3180aeca0 legacy=0x7fb3181db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.137+0000 7fb30e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 1 monclient: found mon.c 2024-03-29T11:19:19.123 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31813b220 con 0x7fb3181df200 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 5 monclient: authenticate success, global_id 4697 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 10 monclient: _renew_subs 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31806ffc0 con 0x7fb3181df200 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 10 monclient: _renew_subs 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31813ac20 con 0x7fb3181df200 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.138+0000 7fb31fae4640 1 librados: init done 2024-03-29T11:19:19.124 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb310001fe0 con 0x7fb3181df200 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7fb310004fb0 con 0x7fb3181df200 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30e7fc640 10 monclient: dump: 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:19.125 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.140+0000 7fb30ffff640 4 set_mon_vals no callback set 2024-03-29T11:19:19.126 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.141+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7fb31001af80 con 0x7fb3181df200 2024-03-29T11:19:19.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.141+0000 7fb31fae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.141+0000 7fb31fae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.141+0000 7fb31fae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.141+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb300005080 con 0x7fb3181df200 2024-03-29T11:19:19.127 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.142+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7fb310084da0 con 0x7fb3181df200 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.142+0000 7fb30e7fc640 10 client.4697.objecter ms_dispatch 0x7fb318000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.142+0000 7fb30e7fc640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.142+0000 7fb30e7fc640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.143+0000 7fb30e7fc640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.143+0000 7fb31d880640 10 monclient: get_auth_request con 0x7fb2fc06dde0 auth_method 0 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.143+0000 7fb30e7fc640 10 client.4697.objecter ms_handle_connect 0x7fb2fc06dde0 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.149+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb310050680 con 0x7fb3181df200 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.149+0000 7fb30e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:19.128 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.149+0000 7fb30e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:19.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.468+0000 7fb31fae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.445 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.468+0000 7fb31fae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.468+0000 7fb31fae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:19.446 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.468+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb300005330 con 0x7fb3181df200 2024-03-29T11:19:19.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.470+0000 7fb30e7fc640 1 -- 172.21.15.105:0/1747568845 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7fb3100525b0 con 0x7fb3181df200 2024-03-29T11:19:19.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.470+0000 7fb30e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.470+0000 7fb30e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:19.447 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:19:19.447 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.474+0000 7fb31fae4640 10 librados: watch_flush enter 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 10 librados: watch_flush exit 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 10 client.4697.objecter successfully canceled tick 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 >> v1:172.21.15.110:6816/1087970004 conn(0x7fb2fc06dde0 legacy=0x7fb2fc0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 10 monclient: shutdown 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 >> v1:172.21.15.105:6790/0 conn(0x7fb3181df200 legacy=0x7fb31813a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 shutdown_connections 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 >> 172.21.15.105:0/1747568845 conn(0x7fb31806b660 msgr2=0x7fb31806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:19.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 shutdown_connections 2024-03-29T11:19:19.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 -- 172.21.15.105:0/1747568845 wait complete. 2024-03-29T11:19:19.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:19.475+0000 7fb31fae4640 1 librados: shutdown 2024-03-29T11:19:19.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:26.275 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.446+0000 7f267359e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.446+0000 7f267359e640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.447+0000 7f267359e640 10 monclient: monmap: 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:19:26.448035+0000 2024-03-29T11:19:26.424 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:19:26.448035+0000 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-b 2024-03-29T11:19:26.425 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.447+0000 7f267359e640 1 Processor -- start 2024-03-29T11:19:26.429 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.448+0000 7f267359e640 1 -- start start 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.448+0000 7f267359e640 10 monclient: init 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient(hunting): picked mon.noname-a con 0x7f266c006870 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient(hunting): picked mon.noname-c con 0x7f266c006c50 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient(hunting): picked mon.noname-b con 0x7f266c0c78e0 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c033a20 con 0x7f266c006870 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c0cb4f0 con 0x7f266c006c50 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c033c70 con 0x7f266c0c78e0 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.430 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267359e640 10 monclient(hunting): authenticate will time out at 1440.725633s 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267259c640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f266c006870 0x7f266c0c4eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:33882/0 (socket says 172.21.15.105:33882) 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.449+0000 7f267259c640 1 -- 172.21.15.105:0/4129181602 learned_addr learned my addr 172.21.15.105:0/4129181602 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522780014 0 0) 0x7f266c033a20 con 0x7f266c006870 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): my global_id is 4620 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c003520 con 0x7f266c006870 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575506244 0 0) 0x7f266c033c70 con 0x7f266c0c78e0 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.431 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient(hunting): my global_id is 4700 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c033a20 con 0x7f266c0c78e0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065194750 0 0) 0x7f266c033a20 con 0x7f266c0c78e0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 >> v1:172.21.15.110:6789/0 conn(0x7f266c006c50 legacy=0x7f266c0cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 >> v1:172.21.15.105:6789/0 conn(0x7f266c006870 legacy=0x7f266c0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 monclient: found mon.noname-b 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c0c7f00 con 0x7f266c0c78e0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.432 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f267404f730 con 0x7f266c0c78e0 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267159a640 10 monclient: dump: 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.433 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267359e640 5 monclient: authenticate success, global_id 4700 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.450+0000 7f267359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.2 v1:172.21.15.105:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2674075f10 con 0x7f266c0c78e0 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 1 -- 172.21.15.105:0/4129181602 <== mon.2 v1:172.21.15.105:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f26740758b0 con 0x7f266c0c78e0 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267159a640 10 monclient: dump: 2024-03-29T11:19:26.434 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.451+0000 7f267359e640 4 set_mon_vals no callback set 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 10 monclient: shutdown 2024-03-29T11:19:26.435 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 1 -- 172.21.15.105:0/4129181602 >> v1:172.21.15.105:6790/0 conn(0x7f266c0c78e0 legacy=0x7f266c0d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 1 -- 172.21.15.105:0/4129181602 shutdown_connections 2024-03-29T11:19:26.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 1 -- 172.21.15.105:0/4129181602 >> 172.21.15.105:0/4129181602 conn(0x7f266c0058f0 msgr2=0x7f266c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 1 -- 172.21.15.105:0/4129181602 shutdown_connections 2024-03-29T11:19:26.436 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.452+0000 7f267359e640 1 -- 172.21.15.105:0/4129181602 wait complete. 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.453+0000 7f267359e640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.453+0000 7f267359e640 10 monclient: monmap: 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:epoch 0 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:last_changed 0.000000 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:created 0.000000 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.noname-a 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.noname-b 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.noname-c 2024-03-29T11:19:26.447 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.454+0000 7f267359e640 1 librados: starting msgr at 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.454+0000 7f267359e640 1 librados: starting objecter 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.454+0000 7f267359e640 1 Processor -- start 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.454+0000 7f267359e640 1 -- start start 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.455+0000 7f267359e640 1 librados: setting wanted keys 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.455+0000 7f267359e640 1 librados: calling monclient init 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.455+0000 7f267359e640 10 monclient: init 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient(hunting): picked mon.noname-c con 0x7f266c006870 addr v1:172.21.15.105:6790/0 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient(hunting): picked mon.noname-b con 0x7f266c0c78e0 addr v1:172.21.15.110:6789/0 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient(hunting): picked mon.noname-a con 0x7f266c0c18d0 addr v1:172.21.15.105:6789/0 2024-03-29T11:19:26.448 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 1 -- --> v1:172.21.15.105:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c02f7b0 con 0x7f266c0c18d0 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c02fa90 con 0x7f266c0c78e0 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 1 -- --> v1:172.21.15.105:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c02feb0 con 0x7f266c006870 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f267359e640 10 monclient(hunting): authenticate will time out at 1440.732748s 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f2672d9d640 1 --1- >> v1:172.21.15.105:6789/0 conn(0x7f266c0c18d0 0x7f266c15d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.105:6789/0 says I am v1:172.21.15.105:34352/0 (socket says 172.21.15.105:34352) 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f2672d9d640 1 -- 172.21.15.105:0/3728398585 learned_addr learned my addr 172.21.15.105:0/3728398585 (peer_addr_for_me v1:172.21.15.105:0/0) 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.456+0000 7f2671d9b640 1 --1- 172.21.15.105:0/3728398585 >> v1:172.21.15.110:6789/0 conn(0x7f266c0c78e0 0x7f266c15ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.105:37618/0 (socket says 172.21.15.105:37618) 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter ms_handle_connect 0x7f266c0c18d0 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter ms_handle_connect 0x7f266c0c78e0 2024-03-29T11:19:26.449 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter ms_handle_connect 0x7f266c006870 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.0 v1:172.21.15.105:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027312143 0 0) 0x7f266c02f7b0 con 0x7f266c0c18d0 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): my global_id is 4623 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2654004eb0 con 0x7f266c0c18d0 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436914910 0 0) 0x7f266c02feb0 con 0x7f266c006870 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.450 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c02f7b0 con 0x7f266c006870 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778179323 0 0) 0x7f266c02fa90 con 0x7f266c0c78e0 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.457+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c02feb0 con 0x7f266c0c78e0 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371674832 0 0) 0x7f266c02f7b0 con 0x7f266c006870 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f266c02fa90 con 0x7f266c006870 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2664001810 con 0x7f266c006870 2024-03-29T11:19:26.451 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.452 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2331706327 0 0) 0x7f266c02feb0 con 0x7f266c0c78e0 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f266c02f7b0 con 0x7f266c0c78e0 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2668001790 con 0x7f266c0c78e0 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.453 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.0 v1:172.21.15.105:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321732917 0 0) 0x7f2654004eb0 con 0x7f266c0c18d0 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f266c02feb0 con 0x7f266c0c18d0 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.0 v1:172.21.15.105:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f267404f400 con 0x7f266c0c18d0 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.454 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1039880766 0 0) 0x7f266c02fa90 con 0x7f266c006870 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 >> v1:172.21.15.110:6789/0 conn(0x7f266c0c78e0 legacy=0x7f266c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 >> v1:172.21.15.105:6789/0 conn(0x7f266c0c18d0 legacy=0x7f266c15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 monclient: found mon.c 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c030040 con 0x7f266c006870 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.455 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f2662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f267359e640 5 monclient: authenticate success, global_id 4703 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f267359e640 10 monclient: _renew_subs 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f267359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.458+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c02e230 con 0x7f266c006870 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f267359e640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f267359e640 10 monclient: _renew_subs 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f267359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c02d630 con 0x7f266c006870 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f267359e640 1 librados: init done 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2664001c20 con 0x7f266c006870 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 763336613 0 0) 0x7f2664004980 con 0x7f266c006870 2024-03-29T11:19:26.456 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2662ffd640 10 monclient: dump: 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:epoch 1 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:fsid 39787f52-b512-4aae-b655-f9112d1c907c 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:last_changed 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:created 2024-03-29T11:07:52.657399+0000 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:election_strategy: 1 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:0: v1:172.21.15.105:6789/0 mon.a 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:1: v1:172.21.15.110:6789/0 mon.b 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:2: v1:172.21.15.105:6790/0 mon.c 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr: 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.459+0000 7f2670d99640 4 set_mon_vals no callback set 2024-03-29T11:19:26.457 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.461+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1111322628 0 0) 0x7f266401a950 con 0x7f266c006870 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2294755049 0 0) 0x7f26640848b0 con 0x7f266c006870 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2662ffd640 10 client.4703.objecter ms_dispatch 0x7f2674000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f267359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f267359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f267359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c038ea0 con 0x7f266c006870 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2662ffd640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2662ffd640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2671d9b640 10 monclient: get_auth_request con 0x7f265406e030 auth_method 0 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.462+0000 7f2662ffd640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.458 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.463+0000 7f2662ffd640 10 client.4703.objecter ms_handle_connect 0x7f265406e030 2024-03-29T11:19:26.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.471+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2664050190 con 0x7f266c006870 2024-03-29T11:19:26.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.471+0000 7f2662ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.459 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.471+0000 7f2662ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.828+0000 7f267359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.828+0000 7f267359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.828+0000 7f267359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.105:6790/0 2024-03-29T11:19:26.806 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.828+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 --> v1:172.21.15.105:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f266c004cd0 con 0x7f266c006870 2024-03-29T11:19:26.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.829+0000 7f2662ffd640 1 -- 172.21.15.105:0/3728398585 <== mon.2 v1:172.21.15.105:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 208438815) 0x7f26640520c0 con 0x7f266c006870 2024-03-29T11:19:26.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.829+0000 7f2662ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.807 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.829+0000 7f2662ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.807 INFO:teuthology.misc.health.smithi105.stdout: 2024-03-29T11:19:26.807 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"39787f52-b512-4aae-b655-f9112d1c907c","created":"2024-03-29T11:08:28.973421+0000","modified":"2024-03-29T11:18:33.910811+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:18:33.910811+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec1dd1d8-e81e-46a7-a39a-63c4e6c93bd2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c5ed9abe-4bfe-4483-99c8-b39191206545","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"34c35504-e154-448d-83c3-102515f1ea0b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e8cf0178-6e7e-43bb-a995-fcd094aa4688","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7b2ae2cc-80b4-4217-a8b8-700093947982","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d31c42e4-e9c2-467f-85a3-ee252451edf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f231f5e6-d6ae-48a0-8a56-52f22a494330","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b03c3d9-2138-4e1b-8a2e-382bd235ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:26.810 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.832+0000 7f267359e640 10 librados: watch_flush enter 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.832+0000 7f267359e640 10 librados: watch_flush exit 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.832+0000 7f267359e640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.832+0000 7f267359e640 10 client.4703.objecter successfully canceled tick 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 >> v1:172.21.15.110:6816/1087970004 conn(0x7f265406e030 legacy=0x7f26540704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 10 monclient: shutdown 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 >> v1:172.21.15.105:6790/0 conn(0x7f266c006870 legacy=0x7f266c0c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 shutdown_connections 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 >> 172.21.15.105:0/3728398585 conn(0x7f266c0058f0 msgr2=0x7f266c039290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 shutdown_connections 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 -- 172.21.15.105:0/3728398585 wait complete. 2024-03-29T11:19:26.811 INFO:teuthology.misc.health.smithi105.stderr:2024-03-29T11:19:26.833+0000 7f267359e640 1 librados: shutdown 2024-03-29T11:19:26.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:26.825 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:19:26.827 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:19:26.828 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:19:26.828 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:26.863 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:19:26.864 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:19:26.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:26.939 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:19:26.939 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:19:26.939 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.010 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:19:27.010 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:19:27.010 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.084 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:19:27.084 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:19:27.084 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.118 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:19:27.118 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:19:27.118 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.195 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:19:27.195 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:19:27.195 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.260 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:19:27.261 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:19:27.261 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.328 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:19:27.328 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:19:27.328 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:19:27.328 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.351 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:19:27.351 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:19:27.352 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.393 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:19:27.393 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:19:27.393 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:19:27.394 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.405 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:19:27.405 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:19:27.405 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.459 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:19:27.459 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:19:27.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:19:27.470 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:19:27.470 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:19:27.470 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:19:27.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:19:27.557 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:31.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:19:31.589 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:31.715 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:19:31.715 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:31.765 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:19:31.765 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:31.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:19:31.815 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:35.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:19:35.387 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:35.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:19:35.473 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:35.532 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:19:35.533 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:35.591 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:19:35.593 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/data/mon.a.tgz 2024-03-29T11:19:35.593 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-03-29T11:19:35.611 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.mKXfPPEySo 2024-03-29T11:19:35.612 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mKXfPPEySo 2024-03-29T11:19:35.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.mKXfPPEySo 2024-03-29T11:19:36.055 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.mKXfPPEySo is 126KB 2024-03-29T11:19:36.106 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.mKXfPPEySo 2024-03-29T11:19:36.122 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/data/mon.c.tgz 2024-03-29T11:19:36.122 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-03-29T11:19:36.177 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.rfFIw3y2dq 2024-03-29T11:19:36.177 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.rfFIw3y2dq 2024-03-29T11:19:36.266 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.rfFIw3y2dq 2024-03-29T11:19:36.351 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.rfFIw3y2dq is 126KB 2024-03-29T11:19:36.373 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.rfFIw3y2dq 2024-03-29T11:19:36.388 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/data/mon.b.tgz 2024-03-29T11:19:36.388 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-29T11:19:36.406 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.KeXicajb42 2024-03-29T11:19:36.407 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KeXicajb42 2024-03-29T11:19:36.521 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.KeXicajb42 2024-03-29T11:19:36.610 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.KeXicajb42 is 126KB 2024-03-29T11:19:36.662 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.KeXicajb42 2024-03-29T11:19:36.677 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:19:36.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:19:36.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:19:36.760 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:19:36.801 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:19:36.801 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:19:36.803 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi105/crash 2024-03-29T11:19:36.804 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:19:36.831 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi110/crash 2024-03-29T11:19:36.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:19:36.868 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:19:36.869 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:19:36.876 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:19:36.938 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34632.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39534.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40550.log: /var/log/ceph/ceph-client.admin.34632.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41177.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34632.log.gz 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39534.log.gz 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41177.log.gz 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-03-29T11:19:36.939 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39339.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40943.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40393.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40198.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40943.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39690.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40393.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40631.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2024-03-29T11:19:36.940 INFO:teuthology.orchestra.run.smithi105.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39690.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40631.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38753.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39495.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38322.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38322.log.gz 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39144.log 2024-03-29T11:19:36.941 INFO:teuthology.orchestra.run.smithi105.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35026.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.35026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39144.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39729.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35026.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39729.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-03-29T11:19:36.942 INFO:teuthology.orchestra.run.smithi105.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39729.log.gz 2024-03-29T11:19:36.943 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-03-29T11:19:36.943 INFO:teuthology.orchestra.run.smithi105.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2024-03-29T11:19:36.943 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-03-29T11:19:36.945 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39105.log: 86.5% 93.5% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40159.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40159.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38244.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41216.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40159.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41216.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:19:36.946 INFO:teuthology.orchestra.run.smithi105.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38244.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39807.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41216.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41294.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39807.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39964.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41294.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38949.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-03-29T11:19:36.947 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38949.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40592.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41099.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39378.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40592.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39378.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41099.log.gz 2024-03-29T11:19:36.948 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39378.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41489.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38831.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41138.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41489.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34927.log 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41489.log.gz 2024-03-29T11:19:36.949 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38519.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41138.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40237.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38519.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34927.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38519.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40237.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38871.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40237.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-03-29T11:19:36.950 INFO:teuthology.orchestra.run.smithi105.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40865.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40433.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38478.log 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-mgr.y.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40865.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28812.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40709.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38478.log: /var/log/ceph/ceph.tmp-client.admin.28812.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28812.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38439.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38478.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41333.log 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40709.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39846.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38439.log.gz 2024-03-29T11:19:36.951 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39222.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41411.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41333.log.gz 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38910.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.0.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz/var/log/ceph/ceph-client.admin.38910.log: 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41060.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38910.log.gz 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40982.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38283.log 2024-03-29T11:19:36.952 INFO:teuthology.orchestra.run.smithi105.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39456.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.38283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41450.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39456.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38283.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39456.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41450.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41255.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41450.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40826.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40354.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41255.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41255.log.gz 2024-03-29T11:19:36.953 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38558.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40354.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40787.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40787.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38636.log: 86.6%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.40787.log.gz 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-03-29T11:19:36.954 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34817.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34817.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34817.log.gz 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34657.log 2024-03-29T11:19:36.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34857.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34777.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34737.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34657.log.gz 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34777.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34857.log.gz 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34777.log.gz 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34697.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.34737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34697.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34897.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34737.log.gz 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34697.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-03-29T11:19:36.962 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34617.log 2024-03-29T11:19:36.963 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34617.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:19:36.963 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34617.log.gz 2024-03-29T11:19:37.013 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:19:48.729 INFO:teuthology.orchestra.run.smithi105.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:19:48.949 INFO:teuthology.orchestra.run.smithi110.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:19:49.134 INFO:teuthology.orchestra.run.smithi110.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:19:49.291 INFO:teuthology.orchestra.run.smithi110.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:19:52.851 INFO:teuthology.orchestra.run.smithi105.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:19:52.874 INFO:teuthology.orchestra.run.smithi105.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:19:53.282 INFO:teuthology.orchestra.run.smithi110.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:19:54.370 INFO:teuthology.orchestra.run.smithi105.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:20:25.925 INFO:teuthology.orchestra.run.smithi105.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:20:35.020 INFO:teuthology.orchestra.run.smithi105.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:20:35.022 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-03-29T11:20:35.022 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m58.135s 2024-03-29T11:20:35.022 INFO:teuthology.orchestra.run.smithi105.stderr:user 2m42.106s 2024-03-29T11:20:35.022 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m5.911s 2024-03-29T11:20:47.243 INFO:teuthology.orchestra.run.smithi110.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi110.stderr:real 1m10.318s 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi110.stderr:user 1m58.690s 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m4.382s 2024-03-29T11:20:47.244 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:20:47.245 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi105/log 2024-03-29T11:20:47.246 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:21:10.889 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi110/log 2024-03-29T11:21:10.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:21:29.002 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:21:29.209 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d47766e7541149d6bb64aa116bdfbdf4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:21:29.212 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:21:29.221 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:21:29.251 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:21:29.252 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:21:29.266 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:21:29.348 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:21:29.358 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:21:29.359 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:21:29.361 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:21:29.375 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:21:29.379 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:21:29.379 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:21:29.380 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-03-29T11:21:29.380 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +428us[ +426us] +/- 81ms 2024-03-29T11:21:29.380 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 -149us[ -149us] +/- 76ms 2024-03-29T11:21:29.380 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv03.front.sepia.ceph.com 3 6 377 43 -251us[ -253us] +/- 76ms 2024-03-29T11:21:29.380 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:29.383 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:21:29.383 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-03-29T11:21:29.383 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +430us[ +426us] +/- 81ms 2024-03-29T11:21:29.383 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -154us[ -157us] +/- 76ms 2024-03-29T11:21:29.384 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -252us[ -256us] +/- 76ms 2024-03-29T11:21:29.384 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:29.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:21:29.393 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:21:29.394 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:21:29.406 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:21:29.491 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:21:29.561 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:21:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:29.593 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:21:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:29.625 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:21:29.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:21:29.646 INFO:teuthology.task.internal:Duration was 1212.403368 seconds 2024-03-29T11:21:29.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:21:29.656 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:21:29.656 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:21:29.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:21:29.701 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:21:29.711 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:21:29.978 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:21:29.978 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-03-29T11:21:29.978 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:21:29.999 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-03-29T11:21:29.999 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:21:30.023 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:21:30.024 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:21:30.042 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:21:30.093 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:21:30.093 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:21:30.110 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:21:30.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:21:30.242 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:21:30.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:21:30.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:21:30.298 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:21:30.308 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:21:30.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:21:30.336 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-03-29T11:21:30.362 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-03-29T11:21:30.401 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:21:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:30.415 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:21:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:30.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:21:30.468 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:21:30.469 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi105 2024-03-29T11:21:30.469 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:21:30.506 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675/remote/smithi110 2024-03-29T11:21:30.506 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:21:30.542 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:21:30.542 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:21:30.544 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:21:30.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:21:30.603 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:21:30.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:21:30.614 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:21:30.614 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:21:30.617 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:21:30.630 INFO:teuthology.orchestra.run.smithi105.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:21 /home/ubuntu/cephtest 2024-03-29T11:21:30.646 INFO:teuthology.orchestra.run.smithi110.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:21 /home/ubuntu/cephtest 2024-03-29T11:21:30.647 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:21:30.657 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:21:30.679 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:21:30.705 DEBUG:teuthology.nuke:shortname: smithi105 2024-03-29T11:21:30.705 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:21:30.729 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:48.700274', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGe2cbzlf1n6FisHkjzCBQV8qL7Y7EdolAUPvoVYE70dLHfL29aFbDMXR0dueLZBqTXP40Xs8fCa9gEC2xFZUw='} 2024-03-29T11:21:30.733 DEBUG:teuthology.nuke:shortname: smithi110 2024-03-29T11:21:30.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:21:30.753 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622675', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:48.701671', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2KushoYANTljforGMe/c4ZdrBe6UZpibQLuMPDjQiS3b4uSst/CWLgNi/sc8NskVETh48c5oBQyYnNV/cicqo='} 2024-03-29T11:21:30.785 INFO:teuthology.orchestra.console.smithi105:Power off 2024-03-29T11:21:30.786 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:21:30.815 INFO:teuthology.orchestra.console.smithi110:Power off 2024-03-29T11:21:30.815 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:21:30.823 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-03-29T11:21:30.823 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:30.836 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-03-29T11:21:30.837 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:30.846 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-03-29T11:21:30.857 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-03-29T11:21:34.847 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:34.858 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:34.958 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-03-29T11:21:34.968 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-03-29T11:21:38.962 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:38.971 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:21:39.074 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-03-29T11:21:39.081 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-03-29T11:21:39.175 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-03-29T11:21:39.182 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-03-29T11:21:39.286 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1212.4033677577972 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d47766e7541149d6bb64aa116bdfbdf4 status: fail success: false 2024-03-29T11:21:39.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:21:39.364 INFO:teuthology.run:FAIL